[==========] Running 14 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 14 tests from AutoRebalancerTest
[ RUN ] AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240502 01:57:56.199622 6615 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.6.117.254:37651,127.6.117.253:46275,127.6.117.252:43201
I20240502 01:57:56.200732 6615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:57:56.201257 6615 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:57:56.208838 6623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:57:56.209090 6621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:57:56.210335 6620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:57:56.210378 6615 server_base.cc:1034] running on GCE node
I20240502 01:57:56.210906 6615 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:57:56.211009 6615 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:57:56.211042 6615 hybrid_clock.cc:648] HybridClock initialized: now 1714615076211040 us; error 0 us; skew 500 ppm
I20240502 01:57:56.212996 6615 webserver.cc:460] Webserver started at http://127.6.117.254:40397/ using document root <none> and password file <none>
I20240502 01:57:56.213538 6615 fs_manager.cc:359] Metadata directory not provided
I20240502 01:57:56.213590 6615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:57:56.213791 6615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:57:56.215500 6615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714615076189642-6615-0/minicluster-data/master-0-root/instance:
uuid: "330ea26ef0a64042a8dbce0db3b4eee1"
format_stamp: "Formatted at 2024-05-02 01:57:56 on dist-test-slave-89cm"
I20240502 01:57:56.219008 6615 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 01:57:56.221236 6628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:56.222327 6615 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 01:57:56.222420 6615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714615076189642-6615-0/minicluster-data/master-0-root
uuid: "330ea26ef0a64042a8dbce0db3b4eee1"
format_stamp: "Formatted at 2024-05-02 01:57:56 on dist-test-slave-89cm"
I20240502 01:57:56.222512 6615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714615076189642-6615-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714615076189642-6615-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714615076189642-6615-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:57:56.239348 6615 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:57:56.240068 6615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:57:56.240208 6615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:57:56.246810 6680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.117.254:37651 every 8 connection(s)
I20240502 01:57:56.246821 6615 rpc_server.cc:307] RPC server started. Bound to: 127.6.117.254:37651
I20240502 01:57:56.247989 6615 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:57:56.249778 6684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:57:56.250375 6681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 01:57:56.250442 6683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:57:56.250981 6686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:57:56.251551 6615 server_base.cc:1034] running on GCE node
I20240502 01:57:56.251703 6615 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:57:56.251739 6615 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:57:56.251760 6615 hybrid_clock.cc:648] HybridClock initialized: now 1714615076251759 us; error 0 us; skew 500 ppm
I20240502 01:57:56.252730 6615 webserver.cc:460] Webserver started at http://127.6.117.253:37943/ using document root <none> and password file <none>
I20240502 01:57:56.252878 6615 fs_manager.cc:359] Metadata directory not provided
I20240502 01:57:56.252924 6615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:57:56.252996 6615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:57:56.253376 6615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714615076189642-6615-0/minicluster-data/master-1-root/instance:
uuid: "c9f6184fabb34268b6285bb1fa04a61d"
format_stamp: "Formatted at 2024-05-02 01:57:56 on dist-test-slave-89cm"
I20240502 01:57:56.254988 6615 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240502 01:57:56.255033 6681 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.117.254" port: 37651 } has no permanent_uuid. Determining permanent_uuid...
I20240502 01:57:56.255838 6691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:56.256074 6615 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 01:57:56.256136 6615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714615076189642-6615-0/minicluster-data/master-1-root
uuid: "c9f6184fabb34268b6285bb1fa04a61d"
format_stamp: "Formatted at 2024-05-02 01:57:56 on dist-test-slave-89cm"
I20240502 01:57:56.256194 6615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714615076189642-6615-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714615076189642-6615-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714615076189642-6615-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:57:56.265559 6681 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.117.253" port: 46275 } has no permanent_uuid. Determining permanent_uuid...
I20240502 01:57:56.265938 6615 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:57:56.266337 6615 kserver.cc:163] Server-wide thread pool size limit: 3276
W20240502 01:57:56.266352 6631 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.117.253:46275: connect: Connection refused (error 111)
W20240502 01:57:56.267129 6681 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.6.117.253:46275: Network error: Client connection negotiation failed: client connection to 127.6.117.253:46275: connect: Connection refused (error 111)
I20240502 01:57:56.270937 6615 rpc_server.cc:307] RPC server started. Bound to: 127.6.117.253:46275
I20240502 01:57:56.270984 6746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.117.253:46275 every 8 connection(s)
I20240502 01:57:56.271554 6615 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 01:57:56.271719 6747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 01:57:56.273038 6749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:57:56.273330 6747 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.117.254" port: 37651 } has no permanent_uuid. Determining permanent_uuid...
W20240502 01:57:56.273398 6750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:57:56.273665 6752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:57:56.273952 6615 server_base.cc:1034] running on GCE node
I20240502 01:57:56.274199 6615 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:57:56.274262 6615 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:57:56.274286 6615 hybrid_clock.cc:648] HybridClock initialized: now 1714615076274285 us; error 0 us; skew 500 ppm
I20240502 01:57:56.275184 6615 webserver.cc:460] Webserver started at http://127.6.117.252:45609/ using document root <none> and password file <none>
I20240502 01:57:56.275317 6615 fs_manager.cc:359] Metadata directory not provided
I20240502 01:57:56.275364 6615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:57:56.275441 6615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:57:56.275786 6615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714615076189642-6615-0/minicluster-data/master-2-root/instance:
uuid: "a0fea32478fa43999e85e0f0cefb9590"
format_stamp: "Formatted at 2024-05-02 01:57:56 on dist-test-slave-89cm"
I20240502 01:57:56.277400 6615 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240502 01:57:56.277980 6747 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.117.253" port: 46275 } has no permanent_uuid. Determining permanent_uuid...
I20240502 01:57:56.278234 6758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:56.278687 6615 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240502 01:57:56.278745 6615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714615076189642-6615-0/minicluster-data/master-2-root
uuid: "a0fea32478fa43999e85e0f0cefb9590"
format_stamp: "Formatted at 2024-05-02 01:57:56 on dist-test-slave-89cm"
I20240502 01:57:56.278807 6615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714615076189642-6615-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714615076189642-6615-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714615076189642-6615-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:57:56.281960 6747 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.117.252" port: 43201 } has no permanent_uuid. Determining permanent_uuid...
W20240502 01:57:56.283000 6747 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.6.117.252:43201: Network error: Client connection negotiation failed: client connection to 127.6.117.252:43201: connect: Connection refused (error 111)
I20240502 01:57:56.289376 6615 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:57:56.289757 6615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:57:56.294178 6615 rpc_server.cc:307] RPC server started. Bound to: 127.6.117.252:43201
I20240502 01:57:56.294232 6812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.117.252:43201 every 8 connection(s)
I20240502 01:57:56.294593 6615 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 01:57:56.295007 6813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:56.296042 6813 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.117.254" port: 37651 } has no permanent_uuid. Determining permanent_uuid...
I20240502 01:57:56.299953 6813 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.117.253" port: 46275 } has no permanent_uuid. Determining permanent_uuid...
I20240502 01:57:56.303206 6813 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.117.252" port: 43201 } has no permanent_uuid. Determining permanent_uuid...
I20240502 01:57:56.308768 6813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0fea32478fa43999e85e0f0cefb9590: Bootstrap starting.
I20240502 01:57:56.311133 6813 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a0fea32478fa43999e85e0f0cefb9590: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:56.311944 6813 log.cc:824] T 00000000000000000000000000000000 P a0fea32478fa43999e85e0f0cefb9590: Log is configured to *not* fsync() on all Append() calls
I20240502 01:57:56.313443 6813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0fea32478fa43999e85e0f0cefb9590: No bootstrap required, opened a new log
I20240502 01:57:56.315912 6813 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0fea32478fa43999e85e0f0cefb9590 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "330ea26ef0a64042a8dbce0db3b4eee1" member_type: VOTER last_known_addr { host: "127.6.117.254" port: 37651 } } peers { permanent_uuid: "c9f6184fabb34268b6285bb1fa04a61d" member_type: VOTER last_known_addr { host: "127.6.117.253" port: 46275 } } peers { permanent_uuid: "a0fea32478fa43999e85e0f0cefb9590" member_type: VOTER last_known_addr { host: "127.6.117.252" port: 43201 } }
I20240502 01:57:56.316067 6813 raft_consensus.cc:374] T 00000000000000000000000000000000 P a0fea32478fa43999e85e0f0cefb9590 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:56.316112 6813 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0fea32478fa43999e85e0f0cefb9590 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0fea32478fa43999e85e0f0cefb9590, State: Initialized, Role: FOLLOWER
I20240502 01:57:56.316421 6681 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.117.253" port: 46275 } attempt: 1
I20240502 01:57:56.316527 6813 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0fea32478fa43999e85e0f0cefb9590 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "330ea26ef0a64042a8dbce0db3b4eee1" member_type: VOTER last_known_addr { host: "127.6.117.254" port: 37651 } } peers { permanent_uuid: "c9f6184fabb34268b6285bb1fa04a61d" member_type: VOTER last_known_addr { host: "127.6.117.253" port: 46275 } } peers { permanent_uuid: "a0fea32478fa43999e85e0f0cefb9590" member_type: VOTER last_known_addr { host: "127.6.117.252" port: 43201 } }
I20240502 01:57:56.317355 6818 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0fea32478fa43999e85e0f0cefb9590 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "330ea26ef0a64042a8dbce0db3b4eee1" member_type: VOTER last_known_addr { host: "127.6.117.254" port: 37651 } } peers { permanent_uuid: "c9f6184fabb34268b6285bb1fa04a61d" member_type: VOTER last_known_addr { host: "127.6.117.253" port: 46275 } } peers { permanent_uuid: "a0fea32478fa43999e85e0f0cefb9590" member_type: VOTER last_known_addr { host: "127.6.117.252" port: 43201 } } }
I20240502 01:57:56.317507 6818 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0fea32478fa43999e85e0f0cefb9590 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 01:57:56.317700 6813 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0fea32478fa43999e85e0f0cefb9590 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 01:57:56.320469 6681 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.117.252" port: 43201 } has no permanent_uuid. Determining permanent_uuid...
W20240502 01:57:56.322793 6829 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a0fea32478fa43999e85e0f0cefb9590: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240502 01:57:56.322857 6829 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240502 01:57:56.325608 6681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 330ea26ef0a64042a8dbce0db3b4eee1: Bootstrap starting.
I20240502 01:57:56.326541 6681 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 330ea26ef0a64042a8dbce0db3b4eee1: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:56.327594 6681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 330ea26ef0a64042a8dbce0db3b4eee1: No bootstrap required, opened a new log
I20240502 01:57:56.328006 6681 raft_consensus.cc:348] T 00000000000000000000000000000000 P 330ea26ef0a64042a8dbce0db3b4eee1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "330ea26ef0a64042a8dbce0db3b4eee1" member_type: VOTER last_known_addr { host: "127.6.117.254" port: 37651 } } peers { permanent_uuid: "c9f6184fabb34268b6285bb1fa04a61d" member_type: VOTER last_known_addr { host: "127.6.117.253" port: 46275 } } peers { permanent_uuid: "a0fea32478fa43999e85e0f0cefb9590" member_type: VOTER last_known_addr { host: "127.6.117.252" port: 43201 } }
I20240502 01:57:56.328104 6681 raft_consensus.cc:374] T 00000000000000000000000000000000 P 330ea26ef0a64042a8dbce0db3b4eee1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:56.328131 6681 raft_consensus.cc:729] T 00000000000000000000000000000000 P 330ea26ef0a64042a8dbce0db3b4eee1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 330ea26ef0a64042a8dbce0db3b4eee1, State: Initialized, Role: FOLLOWER
I20240502 01:57:56.328286 6681 consensus_queue.cc:260] T 00000000000000000000000000000000 P 330ea26ef0a64042a8dbce0db3b4eee1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "330ea26ef0a64042a8dbce0db3b4eee1" member_type: VOTER last_known_addr { host: "127.6.117.254" port: 37651 } } peers { permanent_uuid: "c9f6184fabb34268b6285bb1fa04a61d" member_type: VOTER last_known_addr { host: "127.6.117.253" port: 46275 } } peers { permanent_uuid: "a0fea32478fa43999e85e0f0cefb9590" member_type: VOTER last_known_addr { host: "127.6.117.252" port: 43201 } }
I20240502 01:57:56.328603 6832 sys_catalog.cc:455] T 00000000000000000000000000000000 P 330ea26ef0a64042a8dbce0db3b4eee1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "330ea26ef0a64042a8dbce0db3b4eee1" member_type: VOTER last_known_addr { host: "127.6.117.254" port: 37651 } } peers { permanent_uuid: "c9f6184fabb34268b6285bb1fa04a61d" member_type: VOTER last_known_addr { host: "127.6.117.253" port: 46275 } } peers { permanent_uuid: "a0fea32478fa43999e85e0f0cefb9590" member_type: VOTER last_known_addr { host: "127.6.117.252" port: 43201 } } }
I20240502 01:57:56.328701 6832 sys_catalog.cc:458] T 00000000000000000000000000000000 P 330ea26ef0a64042a8dbce0db3b4eee1 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 01:57:56.328760 6681 sys_catalog.cc:564] T 00000000000000000000000000000000 P 330ea26ef0a64042a8dbce0db3b4eee1 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 01:57:56.330701 6615 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
W20240502 01:57:56.331243 6843 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 330ea26ef0a64042a8dbce0db3b4eee1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240502 01:57:56.331301 6843 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240502 01:57:56.335160 6747 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.117.252" port: 43201 } attempt: 1
I20240502 01:57:56.340322 6747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9f6184fabb34268b6285bb1fa04a61d: Bootstrap starting.
I20240502 01:57:56.341378 6747 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c9f6184fabb34268b6285bb1fa04a61d: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:56.342998 6747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9f6184fabb34268b6285bb1fa04a61d: No bootstrap required, opened a new log
I20240502 01:57:56.343466 6747 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9f6184fabb34268b6285bb1fa04a61d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "330ea26ef0a64042a8dbce0db3b4eee1" member_type: VOTER last_known_addr { host: "127.6.117.254" port: 37651 } } peers { permanent_uuid: "c9f6184fabb34268b6285bb1fa04a61d" member_type: VOTER last_known_addr { host: "127.6.117.253" port: 46275 } } peers { permanent_uuid: "a0fea32478fa43999e85e0f0cefb9590" member_type: VOTER last_known_addr { host: "127.6.117.252" port: 43201 } }
I20240502 01:57:56.343582 6747 raft_consensus.cc:374] T 00000000000000000000000000000000 P c9f6184fabb34268b6285bb1fa04a61d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:56.343621 6747 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9f6184fabb34268b6285bb1fa04a61d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9f6184fabb34268b6285bb1fa04a61d, State: Initialized, Role: FOLLOWER
I20240502 01:57:56.343760 6747 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9f6184fabb34268b6285bb1fa04a61d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "330ea26ef0a64042a8dbce0db3b4eee1" member_type: VOTER last_known_addr { host: "127.6.117.254" port: 37651 } } peers { permanent_uuid: "c9f6184fabb34268b6285bb1fa04a61d" member_type: VOTER last_known_addr { host: "127.6.117.253" port: 46275 } } peers { permanent_uuid: "a0fea32478fa43999e85e0f0cefb9590" member_type: VOTER last_known_addr { host: "127.6.117.252" port: 43201 } }
I20240502 01:57:56.344134 6845 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9f6184fabb34268b6285bb1fa04a61d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "330ea26ef0a64042a8dbce0db3b4eee1" member_type: VOTER last_known_addr { host: "127.6.117.254" port: 37651 } } peers { permanent_uuid: "c9f6184fabb34268b6285bb1fa04a61d" member_type: VOTER last_known_addr { host: "127.6.117.253" port: 46275 } } peers { permanent_uuid: "a0fea32478fa43999e85e0f0cefb9590" member_type: VOTER last_known_addr { host: "127.6.117.252" port: 43201 } } }
I20240502 01:57:56.344239 6747 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9f6184fabb34268b6285bb1fa04a61d [sys.catalog]: configured and running, proceeding with master startup.
I20240502 01:57:56.344246 6845 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9f6184fabb34268b6285bb1fa04a61d [sys.catalog]: This master's current role is: FOLLOWER
I20240502 01:57:56.346326 6615 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
I20240502 01:57:56.346977 6615 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:57:56.347014 6856 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c9f6184fabb34268b6285bb1fa04a61d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240502 01:57:56.347056 6856 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20240502 01:57:56.349133 6857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:57:56.349478 6858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:57:56.349923 6860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:57:56.350288 6615 server_base.cc:1034] running on GCE node
I20240502 01:57:56.350436 6615 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:57:56.350489 6615 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:57:56.350518 6615 hybrid_clock.cc:648] HybridClock initialized: now 1714615076350518 us; error 0 us; skew 500 ppm
I20240502 01:57:56.351500 6615 webserver.cc:460] Webserver started at http://127.6.117.193:42579/ using document root <none> and password file <none>
I20240502 01:57:56.351657 6615 fs_manager.cc:359] Metadata directory not provided
I20240502 01:57:56.351712 6615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:57:56.351778 6615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:57:56.352221 6615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714615076189642-6615-0/minicluster-data/ts-0-root/instance:
uuid: "00f7848b80b344e1968566450a055936"
format_stamp: "Formatted at 2024-05-02 01:57:56 on dist-test-slave-89cm"
I20240502 01:57:56.353929 6615 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240502 01:57:56.354938 6865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:56.355212 6615 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 01:57:56.355271 6615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714615076189642-6615-0/minicluster-data/ts-0-root
uuid: "00f7848b80b344e1968566450a055936"
format_stamp: "Formatted at 2024-05-02 01:57:56 on dist-test-slave-89cm"
I20240502 01:57:56.355338 6615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714615076189642-6615-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714615076189642-6615-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714615076189642-6615-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:57:56.374121 6615 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:57:56.374550 6615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:57:56.374971 6615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:57:56.375792 6615 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:57:56.375851 6615 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:56.375892 6615 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:57:56.375921 6615 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:56.381583 6615 rpc_server.cc:307] RPC server started. Bound to: 127.6.117.193:32875
I20240502 01:57:56.381608 6928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.117.193:32875 every 8 connection(s)
I20240502 01:57:56.382688 6615 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 01:57:56.387982 6929 heartbeater.cc:347] Connected to a master server at 127.6.117.252:43201
I20240502 01:57:56.388123 6929 heartbeater.cc:464] Registering TS with master...
I20240502 01:57:56.388878 6929 heartbeater.cc:511] Master 127.6.117.252:43201 requested a full tablet report, sending...
W20240502 01:57:56.389248 6935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:57:56.390038 6615 server_base.cc:1034] running on GCE node
I20240502 01:57:56.390280 6931 heartbeater.cc:347] Connected to a master server at 127.6.117.253:46275
I20240502 01:57:56.390363 6931 heartbeater.cc:464] Registering TS with master...
I20240502 01:57:56.390537 6931 heartbeater.cc:511] Master 127.6.117.253:46275 requested a full tablet report, sending...
I20240502 01:57:56.390534 6777 ts_manager.cc:194] Registered new tserver with Master: 00f7848b80b344e1968566450a055936 (127.6.117.193:32875)
I20240502 01:57:56.390633 6930 heartbeater.cc:347] Connected to a master server at 127.6.117.254:37651
I20240502 01:57:56.390734 6930 heartbeater.cc:464] Registering TS with master...
I20240502 01:57:56.391044 6930 heartbeater.cc:511] Master 127.6.117.254:37651 requested a full tablet report, sending...
I20240502 01:57:56.391404 6711 ts_manager.cc:194] Registered new tserver with Master: 00f7848b80b344e1968566450a055936 (127.6.117.193:32875)
W20240502 01:57:56.391788 6936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:57:56.391831 6644 ts_manager.cc:194] Registered new tserver with Master: 00f7848b80b344e1968566450a055936 (127.6.117.193:32875)
W20240502 01:57:56.392050 6938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:57:56.392326 6615 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:57:56.392374 6615 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:57:56.392388 6615 hybrid_clock.cc:648] HybridClock initialized: now 1714615076392388 us; error 0 us; skew 500 ppm
I20240502 01:57:56.393250 6615 webserver.cc:460] Webserver started at http://127.6.117.194:45327/ using document root <none> and password file <none>
I20240502 01:57:56.393393 6615 fs_manager.cc:359] Metadata directory not provided
I20240502 01:57:56.393437 6615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:57:56.393504 6615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:57:56.393854 6615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714615076189642-6615-0/minicluster-data/ts-1-root/instance:
uuid: "1266b98ba1bc4d8097ba226f05804bdc"
format_stamp: "Formatted at 2024-05-02 01:57:56 on dist-test-slave-89cm"
I20240502 01:57:56.395202 6615 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240502 01:57:56.396024 6944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:56.396313 6615 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240502 01:57:56.396384 6615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714615076189642-6615-0/minicluster-data/ts-1-root
uuid: "1266b98ba1bc4d8097ba226f05804bdc"
format_stamp: "Formatted at 2024-05-02 01:57:56 on dist-test-slave-89cm"
I20240502 01:57:56.396453 6615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714615076189642-6615-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714615076189642-6615-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714615076189642-6615-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:57:56.438102 6615 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:57:56.438568 6615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:57:56.438920 6615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:57:56.439456 6615 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:57:56.439507 6615 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:56.439548 6615 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:57:56.439567 6615 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:56.445462 6615 rpc_server.cc:307] RPC server started. Bound to: 127.6.117.194:39363
I20240502 01:57:56.445559 7007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.117.194:39363 every 8 connection(s)
I20240502 01:57:56.446491 6615 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:57:56.449218 7015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:57:56.450001 7014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:57:56.450170 7017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:57:56.451092 6615 server_base.cc:1034] running on GCE node
I20240502 01:57:56.451283 6615 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:57:56.451341 6615 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:57:56.451366 6615 hybrid_clock.cc:648] HybridClock initialized: now 1714615076451366 us; error 0 us; skew 500 ppm
I20240502 01:57:56.451709 7008 heartbeater.cc:347] Connected to a master server at 127.6.117.252:43201
I20240502 01:57:56.451798 7008 heartbeater.cc:464] Registering TS with master...
I20240502 01:57:56.451936 7009 heartbeater.cc:347] Connected to a master server at 127.6.117.254:37651
I20240502 01:57:56.452015 7008 heartbeater.cc:511] Master 127.6.117.252:43201 requested a full tablet report, sending...
I20240502 01:57:56.452014 7009 heartbeater.cc:464] Registering TS with master...
I20240502 01:57:56.452298 7009 heartbeater.cc:511] Master 127.6.117.254:37651 requested a full tablet report, sending...
I20240502 01:57:56.452704 6777 ts_manager.cc:194] Registered new tserver with Master: 1266b98ba1bc4d8097ba226f05804bdc (127.6.117.194:39363)
I20240502 01:57:56.452726 6615 webserver.cc:460] Webserver started at http://127.6.117.195:37499/ using document root <none> and password file <none>
I20240502 01:57:56.452880 6615 fs_manager.cc:359] Metadata directory not provided
I20240502 01:57:56.452948 6644 ts_manager.cc:194] Registered new tserver with Master: 1266b98ba1bc4d8097ba226f05804bdc (127.6.117.194:39363)
I20240502 01:57:56.452968 6615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:57:56.453092 6615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:57:56.453519 6615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714615076189642-6615-0/minicluster-data/ts-2-root/instance:
uuid: "7248976624e2410f9d6ffd6ceb10b526"
format_stamp: "Formatted at 2024-05-02 01:57:56 on dist-test-slave-89cm"
I20240502 01:57:56.454982 7010 heartbeater.cc:347] Connected to a master server at 127.6.117.253:46275
I20240502 01:57:56.455062 7010 heartbeater.cc:464] Registering TS with master...
I20240502 01:57:56.455250 7010 heartbeater.cc:511] Master 127.6.117.253:46275 requested a full tablet report, sending...
I20240502 01:57:56.455345 6615 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20240502 01:57:56.456171 6711 ts_manager.cc:194] Registered new tserver with Master: 1266b98ba1bc4d8097ba226f05804bdc (127.6.117.194:39363)
I20240502 01:57:56.456487 7023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:56.456758 6615 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 01:57:56.456820 6615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714615076189642-6615-0/minicluster-data/ts-2-root
uuid: "7248976624e2410f9d6ffd6ceb10b526"
format_stamp: "Formatted at 2024-05-02 01:57:56 on dist-test-slave-89cm"
I20240502 01:57:56.456873 6615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714615076189642-6615-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714615076189642-6615-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714615076189642-6615-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:57:56.485644 6615 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:57:56.486102 6615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:57:56.486423 6615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:57:56.486912 6615 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:57:56.486954 6615 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:56.486999 6615 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:57:56.487025 6615 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20240502 01:57:56.491402 6634 tablet.cc:2369] T 00000000000000000000000000000000 P 330ea26ef0a64042a8dbce0db3b4eee1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:57:56.491902 6615 rpc_server.cc:307] RPC server started. Bound to: 127.6.117.195:45111
I20240502 01:57:56.491991 7086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.117.195:45111 every 8 connection(s)
I20240502 01:57:56.497407 7088 heartbeater.cc:347] Connected to a master server at 127.6.117.254:37651
I20240502 01:57:56.497448 7087 heartbeater.cc:347] Connected to a master server at 127.6.117.252:43201
I20240502 01:57:56.497505 7088 heartbeater.cc:464] Registering TS with master...
I20240502 01:57:56.497731 7088 heartbeater.cc:511] Master 127.6.117.254:37651 requested a full tablet report, sending...
I20240502 01:57:56.497505 7087 heartbeater.cc:464] Registering TS with master...
I20240502 01:57:56.498224 7087 heartbeater.cc:511] Master 127.6.117.252:43201 requested a full tablet report, sending...
I20240502 01:57:56.498574 6644 ts_manager.cc:194] Registered new tserver with Master: 7248976624e2410f9d6ffd6ceb10b526 (127.6.117.195:45111)
I20240502 01:57:56.498822 6777 ts_manager.cc:194] Registered new tserver with Master: 7248976624e2410f9d6ffd6ceb10b526 (127.6.117.195:45111)
I20240502 01:57:56.499269 7089 heartbeater.cc:347] Connected to a master server at 127.6.117.253:46275
I20240502 01:57:56.499341 7089 heartbeater.cc:464] Registering TS with master...
I20240502 01:57:56.499502 7089 heartbeater.cc:511] Master 127.6.117.253:46275 requested a full tablet report, sending...
I20240502 01:57:56.500025 6711 ts_manager.cc:194] Registered new tserver with Master: 7248976624e2410f9d6ffd6ceb10b526 (127.6.117.195:45111)
I20240502 01:57:56.500720 6615 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.008241376s
I20240502 01:57:56.543592 6818 raft_consensus.cc:482] T 00000000000000000000000000000000 P a0fea32478fa43999e85e0f0cefb9590 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:57:56.543824 6818 raft_consensus.cc:504] T 00000000000000000000000000000000 P a0fea32478fa43999e85e0f0cefb9590 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "330ea26ef0a64042a8dbce0db3b4eee1" member_type: VOTER last_known_addr { host: "127.6.117.254" port: 37651 } } peers { permanent_uuid: "c9f6184fabb34268b6285bb1fa04a61d" member_type: VOTER last_known_addr { host: "127.6.117.253" port: 46275 } } peers { permanent_uuid: "a0fea32478fa43999e85e0f0cefb9590" member_type: VOTER last_known_addr { host: "127.6.117.252" port: 43201 } }
I20240502 01:57:56.544791 6818 leader_election.cc:288] T 00000000000000000000000000000000 P a0fea32478fa43999e85e0f0cefb9590 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 330ea26ef0a64042a8dbce0db3b4eee1 (127.6.117.254:37651), c9f6184fabb34268b6285bb1fa04a61d (127.6.117.253:46275)
I20240502 01:57:56.545295 6655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a0fea32478fa43999e85e0f0cefb9590" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "330ea26ef0a64042a8dbce0db3b4eee1" is_pre_election: true
I20240502 01:57:56.545303 6721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a0fea32478fa43999e85e0f0cefb9590" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f6184fabb34268b6285bb1fa04a61d" is_pre_election: true
I20240502 01:57:56.545508 6721 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c9f6184fabb34268b6285bb1fa04a61d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0fea32478fa43999e85e0f0cefb9590 in term 0.
I20240502 01:57:56.545508 6655 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 330ea26ef0a64042a8dbce0db3b4eee1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0fea32478fa43999e85e0f0cefb9590 in term 0.
I20240502 01:57:56.545895 6762 leader_election.cc:302] T 00000000000000000000000000000000 P a0fea32478fa43999e85e0f0cefb9590 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0fea32478fa43999e85e0f0cefb9590, c9f6184fabb34268b6285bb1fa04a61d; no voters:
I20240502 01:57:56.546145 6818 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a0fea32478fa43999e85e0f0cefb9590 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:57:56.546214 6818 raft_consensus.cc:482] T 00000000000000000000000000000000 P a0fea32478fa43999e85e0f0cefb9590 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:57:56.546247 6818 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0fea32478fa43999e85e0f0cefb9590 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:56.547116 6818 raft_consensus.cc:504] T 00000000000000000000000000000000 P a0fea32478fa43999e85e0f0cefb9590 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "330ea26ef0a64042a8dbce0db3b4eee1" member_type: VOTER last_known_addr { host: "127.6.117.254" port: 37651 } } peers { permanent_uuid: "c9f6184fabb34268b6285bb1fa04a61d" member_type: VOTER last_known_addr { host: "127.6.117.253" port: 46275 } } peers { permanent_uuid: "a0fea32478fa43999e85e0f0cefb9590" member_type: VOTER last_known_addr { host: "127.6.117.252" port: 43201 } }
I20240502 01:57:56.547430 6818 leader_election.cc:288] T 00000000000000000000000000000000 P a0fea32478fa43999e85e0f0cefb9590 [CANDIDATE]: Term 1 election: Requested vote from peers 330ea26ef0a64042a8dbce0db3b4eee1 (127.6.117.254:37651), c9f6184fabb34268b6285bb1fa04a61d (127.6.117.253:46275)
I20240502 01:57:56.547710 6721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a0fea32478fa43999e85e0f0cefb9590" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f6184fabb34268b6285bb1fa04a61d"
I20240502 01:57:56.547710 6655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a0fea32478fa43999e85e0f0cefb9590" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "330ea26ef0a64042a8dbce0db3b4eee1"
I20240502 01:57:56.547848 6655 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 330ea26ef0a64042a8dbce0db3b4eee1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:56.547863 6721 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9f6184fabb34268b6285bb1fa04a61d [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:56.548774 6655 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 330ea26ef0a64042a8dbce0db3b4eee1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0fea32478fa43999e85e0f0cefb9590 in term 1.
I20240502 01:57:56.548774 6721 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c9f6184fabb34268b6285bb1fa04a61d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0fea32478fa43999e85e0f0cefb9590 in term 1.
I20240502 01:57:56.549072 6762 leader_election.cc:302] T 00000000000000000000000000000000 P a0fea32478fa43999e85e0f0cefb9590 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0fea32478fa43999e85e0f0cefb9590, c9f6184fabb34268b6285bb1fa04a61d; no voters:
I20240502 01:57:56.549232 6818 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a0fea32478fa43999e85e0f0cefb9590 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:57:56.549463 6818 raft_consensus.cc:686] T 00000000000000000000000000000000 P a0fea32478fa43999e85e0f0cefb9590 [term 1 LEADER]: Becoming Leader. State: Replica: a0fea32478fa43999e85e0f0cefb9590, State: Running, Role: LEADER
I20240502 01:57:56.549714 6818 consensus_queue.cc:237] T 00000000000000000000000000000000 P a0fea32478fa43999e85e0f0cefb9590 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "330ea26ef0a64042a8dbce0db3b4eee1" member_type: VOTER last_known_addr { host: "127.6.117.254" port: 37651 } } peers { permanent_uuid: "c9f6184fabb34268b6285bb1fa04a61d" member_type: VOTER last_known_addr { host: "127.6.117.253" port: 46275 } } peers { permanent_uuid: "a0fea32478fa43999e85e0f0cefb9590" member_type: VOTER last_known_addr { host: "127.6.117.252" port: 43201 } }
I20240502 01:57:56.551385 7095 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0fea32478fa43999e85e0f0cefb9590 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0fea32478fa43999e85e0f0cefb9590. Latest consensus state: current_term: 1 leader_uuid: "a0fea32478fa43999e85e0f0cefb9590" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "330ea26ef0a64042a8dbce0db3b4eee1" member_type: VOTER last_known_addr { host: "127.6.117.254" port: 37651 } } peers { permanent_uuid: "c9f6184fabb34268b6285bb1fa04a61d" member_type: VOTER last_known_addr { host: "127.6.117.253" port: 46275 } } peers { permanent_uuid: "a0fea32478fa43999e85e0f0cefb9590" member_type: VOTER last_known_addr { host: "127.6.117.252" port: 43201 } } }
I20240502 01:57:56.551499 7095 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0fea32478fa43999e85e0f0cefb9590 [sys.catalog]: This master's current role is: LEADER
I20240502 01:57:56.551862 7097 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 01:57:56.552803 7097 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 01:57:56.557122 6721 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c9f6184fabb34268b6285bb1fa04a61d [term 1 FOLLOWER]: Refusing update from remote peer a0fea32478fa43999e85e0f0cefb9590: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:57:56.557122 6655 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 330ea26ef0a64042a8dbce0db3b4eee1 [term 1 FOLLOWER]: Refusing update from remote peer a0fea32478fa43999e85e0f0cefb9590: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:57:56.557627 7095 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a0fea32478fa43999e85e0f0cefb9590 [LEADER]: Connected to new peer: Peer: permanent_uuid: "330ea26ef0a64042a8dbce0db3b4eee1" member_type: VOTER last_known_addr { host: "127.6.117.254" port: 37651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:57:56.557780 6818 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a0fea32478fa43999e85e0f0cefb9590 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f6184fabb34268b6285bb1fa04a61d" member_type: VOTER last_known_addr { host: "127.6.117.253" port: 46275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:57:56.559341 6832 sys_catalog.cc:455] T 00000000000000000000000000000000 P 330ea26ef0a64042a8dbce0db3b4eee1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0fea32478fa43999e85e0f0cefb9590. Latest consensus state: current_term: 1 leader_uuid: "a0fea32478fa43999e85e0f0cefb9590" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "330ea26ef0a64042a8dbce0db3b4eee1" member_type: VOTER last_known_addr { host: "127.6.117.254" port: 37651 } } peers { permanent_uuid: "c9f6184fabb34268b6285bb1fa04a61d" member_type: VOTER last_known_addr { host: "127.6.117.253" port: 46275 } } peers { permanent_uuid: "a0fea32478fa43999e85e0f0cefb9590" member_type: VOTER last_known_addr { host: "127.6.117.252" port: 43201 } } }
I20240502 01:57:56.559360 6845 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9f6184fabb34268b6285bb1fa04a61d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0fea32478fa43999e85e0f0cefb9590. Latest consensus state: current_term: 1 leader_uuid: "a0fea32478fa43999e85e0f0cefb9590" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "330ea26ef0a64042a8dbce0db3b4eee1" member_type: VOTER last_known_addr { host: "127.6.117.254" port: 37651 } } peers { permanent_uuid: "c9f6184fabb34268b6285bb1fa04a61d" member_type: VOTER last_known_addr { host: "127.6.117.253" port: 46275 } } peers { permanent_uuid: "a0fea32478fa43999e85e0f0cefb9590" member_type: VOTER last_known_addr { host: "127.6.117.252" port: 43201 } } }
I20240502 01:57:56.559461 6832 sys_catalog.cc:458] T 00000000000000000000000000000000 P 330ea26ef0a64042a8dbce0db3b4eee1 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 01:57:56.559473 6845 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9f6184fabb34268b6285bb1fa04a61d [sys.catalog]: This master's current role is: FOLLOWER
I20240502 01:57:56.561447 6845 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9f6184fabb34268b6285bb1fa04a61d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a0fea32478fa43999e85e0f0cefb9590" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "330ea26ef0a64042a8dbce0db3b4eee1" member_type: VOTER last_known_addr { host: "127.6.117.254" port: 37651 } } peers { permanent_uuid: "c9f6184fabb34268b6285bb1fa04a61d" member_type: VOTER last_known_addr { host: "127.6.117.253" port: 46275 } } peers { permanent_uuid: "a0fea32478fa43999e85e0f0cefb9590" member_type: VOTER last_known_addr { host: "127.6.117.252" port: 43201 } } }
I20240502 01:57:56.561589 6845 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9f6184fabb34268b6285bb1fa04a61d [sys.catalog]: This master's current role is: FOLLOWER
I20240502 01:57:56.561463 6832 sys_catalog.cc:455] T 00000000000000000000000000000000 P 330ea26ef0a64042a8dbce0db3b4eee1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a0fea32478fa43999e85e0f0cefb9590" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "330ea26ef0a64042a8dbce0db3b4eee1" member_type: VOTER last_known_addr { host: "127.6.117.254" port: 37651 } } peers { permanent_uuid: "c9f6184fabb34268b6285bb1fa04a61d" member_type: VOTER last_known_addr { host: "127.6.117.253" port: 46275 } } peers { permanent_uuid: "a0fea32478fa43999e85e0f0cefb9590" member_type: VOTER last_known_addr { host: "127.6.117.252" port: 43201 } } }
I20240502 01:57:56.561748 6832 sys_catalog.cc:458] T 00000000000000000000000000000000 P 330ea26ef0a64042a8dbce0db3b4eee1 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 01:57:56.562183 6818 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0fea32478fa43999e85e0f0cefb9590 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a0fea32478fa43999e85e0f0cefb9590" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "330ea26ef0a64042a8dbce0db3b4eee1" member_type: VOTER last_known_addr { host: "127.6.117.254" port: 37651 } } peers { permanent_uuid: "c9f6184fabb34268b6285bb1fa04a61d" member_type: VOTER last_known_addr { host: "127.6.117.253" port: 46275 } } peers { permanent_uuid: "a0fea32478fa43999e85e0f0cefb9590" member_type: VOTER last_known_addr { host: "127.6.117.252" port: 43201 } } }
I20240502 01:57:56.562206 7095 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0fea32478fa43999e85e0f0cefb9590 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a0fea32478fa43999e85e0f0cefb9590" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "330ea26ef0a64042a8dbce0db3b4eee1" member_type: VOTER last_known_addr { host: "127.6.117.254" port: 37651 } } peers { permanent_uuid: "c9f6184fabb34268b6285bb1fa04a61d" member_type: VOTER last_known_addr { host: "127.6.117.253" port: 46275 } } peers { permanent_uuid: "a0fea32478fa43999e85e0f0cefb9590" member_type: VOTER last_known_addr { host: "127.6.117.252" port: 43201 } } }
I20240502 01:57:56.562273 6818 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0fea32478fa43999e85e0f0cefb9590 [sys.catalog]: This master's current role is: LEADER
I20240502 01:57:56.562292 7095 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0fea32478fa43999e85e0f0cefb9590 [sys.catalog]: This master's current role is: LEADER
I20240502 01:57:56.562422 7099 mvcc.cc:205] Tried to move back new op lower bound from 7023063353572159488 to 7023063353549967360. Current Snapshot: MvccSnapshot[applied={T|T < 7023063353572159488}]
I20240502 01:57:56.562955 7097 catalog_manager.cc:1347] Generated new cluster ID: 495efaf2765546668e0efe2be04c875a
I20240502 01:57:56.563014 7097 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 01:57:56.576514 7097 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 01:57:56.577404 7097 catalog_manager.cc:1504] Loading token signing keys...
I20240502 01:57:56.583343 7097 catalog_manager.cc:5857] T 00000000000000000000000000000000 P a0fea32478fa43999e85e0f0cefb9590: Generated new TSK 0
I20240502 01:57:56.584308 7097 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 01:57:57.332108 6843 catalog_manager.cc:1259] Loaded cluster ID: 495efaf2765546668e0efe2be04c875a
I20240502 01:57:57.332197 6843 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 330ea26ef0a64042a8dbce0db3b4eee1: loading cluster ID for follower catalog manager: success
I20240502 01:57:57.333567 6843 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 330ea26ef0a64042a8dbce0db3b4eee1: acquiring CA information for follower catalog manager: success
I20240502 01:57:57.334245 6843 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 330ea26ef0a64042a8dbce0db3b4eee1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240502 01:57:57.347865 6856 catalog_manager.cc:1259] Loaded cluster ID: 495efaf2765546668e0efe2be04c875a
I20240502 01:57:57.347952 6856 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c9f6184fabb34268b6285bb1fa04a61d: loading cluster ID for follower catalog manager: success
I20240502 01:57:57.349200 6856 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c9f6184fabb34268b6285bb1fa04a61d: acquiring CA information for follower catalog manager: success
I20240502 01:57:57.349777 6856 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c9f6184fabb34268b6285bb1fa04a61d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240502 01:57:57.393764 6777 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48294
I20240502 01:57:57.454707 6777 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48308
I20240502 01:57:57.500875 6777 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48318
I20240502 01:57:58.319713 6828 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:57:58.395040 6929 heartbeater.cc:503] Master 127.6.117.252:43201 was elected leader, sending a full tablet report...
I20240502 01:57:58.455612 7008 heartbeater.cc:503] Master 127.6.117.252:43201 was elected leader, sending a full tablet report...
I20240502 01:57:58.501776 7087 heartbeater.cc:503] Master 127.6.117.252:43201 was elected leader, sending a full tablet report...
I20240502 01:57:58.655946 6615 test_util.cc:268] Using random seed: -290416700
I20240502 01:57:58.668834 6777 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48328:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 01:57:58.686298 6892 tablet_service.cc:1467] Processing CreateTablet for tablet a266cf6c4b8d4004808d091e48312078 (DEFAULT_TABLE table=test-workload [id=ca814f44d7d942a48bd822694df5eeff]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240502 01:57:58.686295 6972 tablet_service.cc:1467] Processing CreateTablet for tablet 3fec98c93c4f4fd6999c51b10f25d054 (DEFAULT_TABLE table=test-workload [id=ca814f44d7d942a48bd822694df5eeff]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240502 01:57:58.686287 6893 tablet_service.cc:1467] Processing CreateTablet for tablet c5ee2d565791431387fa0a0c5bc3a634 (DEFAULT_TABLE table=test-workload [id=ca814f44d7d942a48bd822694df5eeff]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240502 01:57:58.686295 7051 tablet_service.cc:1467] Processing CreateTablet for tablet 8263a9836e514c0d9c0504172816d961 (DEFAULT_TABLE table=test-workload [id=ca814f44d7d942a48bd822694df5eeff]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240502 01:57:58.686805 7051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8263a9836e514c0d9c0504172816d961. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:58.686793 6892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a266cf6c4b8d4004808d091e48312078. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:58.686795 6972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fec98c93c4f4fd6999c51b10f25d054. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:58.689087 7135 tablet_bootstrap.cc:492] T 3fec98c93c4f4fd6999c51b10f25d054 P 1266b98ba1bc4d8097ba226f05804bdc: Bootstrap starting.
I20240502 01:57:58.689133 6893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5ee2d565791431387fa0a0c5bc3a634. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:58.689651 7137 tablet_bootstrap.cc:492] T a266cf6c4b8d4004808d091e48312078 P 00f7848b80b344e1968566450a055936: Bootstrap starting.
I20240502 01:57:58.689997 7135 tablet_bootstrap.cc:654] T 3fec98c93c4f4fd6999c51b10f25d054 P 1266b98ba1bc4d8097ba226f05804bdc: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:58.690325 7137 tablet_bootstrap.cc:654] T a266cf6c4b8d4004808d091e48312078 P 00f7848b80b344e1968566450a055936: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:58.690881 7136 tablet_bootstrap.cc:492] T 8263a9836e514c0d9c0504172816d961 P 7248976624e2410f9d6ffd6ceb10b526: Bootstrap starting.
I20240502 01:57:58.691110 7135 tablet_bootstrap.cc:492] T 3fec98c93c4f4fd6999c51b10f25d054 P 1266b98ba1bc4d8097ba226f05804bdc: No bootstrap required, opened a new log
I20240502 01:57:58.691192 7135 ts_tablet_manager.cc:1367] T 3fec98c93c4f4fd6999c51b10f25d054 P 1266b98ba1bc4d8097ba226f05804bdc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:57:58.691501 7135 raft_consensus.cc:348] T 3fec98c93c4f4fd6999c51b10f25d054 P 1266b98ba1bc4d8097ba226f05804bdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1266b98ba1bc4d8097ba226f05804bdc" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 39363 } }
I20240502 01:57:58.691577 7136 tablet_bootstrap.cc:654] T 8263a9836e514c0d9c0504172816d961 P 7248976624e2410f9d6ffd6ceb10b526: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:58.691583 7135 raft_consensus.cc:374] T 3fec98c93c4f4fd6999c51b10f25d054 P 1266b98ba1bc4d8097ba226f05804bdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:58.691699 7135 raft_consensus.cc:729] T 3fec98c93c4f4fd6999c51b10f25d054 P 1266b98ba1bc4d8097ba226f05804bdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1266b98ba1bc4d8097ba226f05804bdc, State: Initialized, Role: FOLLOWER
I20240502 01:57:58.691865 7135 consensus_queue.cc:260] T 3fec98c93c4f4fd6999c51b10f25d054 P 1266b98ba1bc4d8097ba226f05804bdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1266b98ba1bc4d8097ba226f05804bdc" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 39363 } }
I20240502 01:57:58.691965 7135 raft_consensus.cc:388] T 3fec98c93c4f4fd6999c51b10f25d054 P 1266b98ba1bc4d8097ba226f05804bdc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:57:58.692016 7135 raft_consensus.cc:482] T 3fec98c93c4f4fd6999c51b10f25d054 P 1266b98ba1bc4d8097ba226f05804bdc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:57:58.692065 7135 raft_consensus.cc:3037] T 3fec98c93c4f4fd6999c51b10f25d054 P 1266b98ba1bc4d8097ba226f05804bdc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:58.692091 7137 tablet_bootstrap.cc:492] T a266cf6c4b8d4004808d091e48312078 P 00f7848b80b344e1968566450a055936: No bootstrap required, opened a new log
I20240502 01:57:58.692166 7137 ts_tablet_manager.cc:1367] T a266cf6c4b8d4004808d091e48312078 P 00f7848b80b344e1968566450a055936: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:57:58.692541 7137 raft_consensus.cc:348] T a266cf6c4b8d4004808d091e48312078 P 00f7848b80b344e1968566450a055936 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00f7848b80b344e1968566450a055936" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 32875 } }
I20240502 01:57:58.692631 7137 raft_consensus.cc:374] T a266cf6c4b8d4004808d091e48312078 P 00f7848b80b344e1968566450a055936 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:58.692662 7137 raft_consensus.cc:729] T a266cf6c4b8d4004808d091e48312078 P 00f7848b80b344e1968566450a055936 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00f7848b80b344e1968566450a055936, State: Initialized, Role: FOLLOWER
I20240502 01:57:58.692767 7137 consensus_queue.cc:260] T a266cf6c4b8d4004808d091e48312078 P 00f7848b80b344e1968566450a055936 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00f7848b80b344e1968566450a055936" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 32875 } }
I20240502 01:57:58.692802 7136 tablet_bootstrap.cc:492] T 8263a9836e514c0d9c0504172816d961 P 7248976624e2410f9d6ffd6ceb10b526: No bootstrap required, opened a new log
I20240502 01:57:58.692898 7136 ts_tablet_manager.cc:1367] T 8263a9836e514c0d9c0504172816d961 P 7248976624e2410f9d6ffd6ceb10b526: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:57:58.692898 7135 raft_consensus.cc:504] T 3fec98c93c4f4fd6999c51b10f25d054 P 1266b98ba1bc4d8097ba226f05804bdc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1266b98ba1bc4d8097ba226f05804bdc" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 39363 } }
I20240502 01:57:58.693065 7135 leader_election.cc:302] T 3fec98c93c4f4fd6999c51b10f25d054 P 1266b98ba1bc4d8097ba226f05804bdc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1266b98ba1bc4d8097ba226f05804bdc; no voters:
I20240502 01:57:58.693239 7135 leader_election.cc:288] T 3fec98c93c4f4fd6999c51b10f25d054 P 1266b98ba1bc4d8097ba226f05804bdc [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:57:58.693311 7141 raft_consensus.cc:2781] T 3fec98c93c4f4fd6999c51b10f25d054 P 1266b98ba1bc4d8097ba226f05804bdc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:57:58.693316 7136 raft_consensus.cc:348] T 8263a9836e514c0d9c0504172816d961 P 7248976624e2410f9d6ffd6ceb10b526 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7248976624e2410f9d6ffd6ceb10b526" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 45111 } }
I20240502 01:57:58.693428 7136 raft_consensus.cc:374] T 8263a9836e514c0d9c0504172816d961 P 7248976624e2410f9d6ffd6ceb10b526 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:58.693467 7136 raft_consensus.cc:729] T 8263a9836e514c0d9c0504172816d961 P 7248976624e2410f9d6ffd6ceb10b526 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7248976624e2410f9d6ffd6ceb10b526, State: Initialized, Role: FOLLOWER
I20240502 01:57:58.693501 7141 raft_consensus.cc:686] T 3fec98c93c4f4fd6999c51b10f25d054 P 1266b98ba1bc4d8097ba226f05804bdc [term 1 LEADER]: Becoming Leader. State: Replica: 1266b98ba1bc4d8097ba226f05804bdc, State: Running, Role: LEADER
I20240502 01:57:58.693542 7135 ts_tablet_manager.cc:1398] T 3fec98c93c4f4fd6999c51b10f25d054 P 1266b98ba1bc4d8097ba226f05804bdc: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 01:57:58.693580 7136 consensus_queue.cc:260] T 8263a9836e514c0d9c0504172816d961 P 7248976624e2410f9d6ffd6ceb10b526 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7248976624e2410f9d6ffd6ceb10b526" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 45111 } }
I20240502 01:57:58.693657 7136 raft_consensus.cc:388] T 8263a9836e514c0d9c0504172816d961 P 7248976624e2410f9d6ffd6ceb10b526 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:57:58.693625 7141 consensus_queue.cc:237] T 3fec98c93c4f4fd6999c51b10f25d054 P 1266b98ba1bc4d8097ba226f05804bdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1266b98ba1bc4d8097ba226f05804bdc" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 39363 } }
I20240502 01:57:58.693687 7136 raft_consensus.cc:482] T 8263a9836e514c0d9c0504172816d961 P 7248976624e2410f9d6ffd6ceb10b526 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:57:58.693720 7136 raft_consensus.cc:3037] T 8263a9836e514c0d9c0504172816d961 P 7248976624e2410f9d6ffd6ceb10b526 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:58.693771 7137 raft_consensus.cc:388] T a266cf6c4b8d4004808d091e48312078 P 00f7848b80b344e1968566450a055936 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:57:58.693817 7137 raft_consensus.cc:482] T a266cf6c4b8d4004808d091e48312078 P 00f7848b80b344e1968566450a055936 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:57:58.693852 7137 raft_consensus.cc:3037] T a266cf6c4b8d4004808d091e48312078 P 00f7848b80b344e1968566450a055936 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:58.694492 7136 raft_consensus.cc:504] T 8263a9836e514c0d9c0504172816d961 P 7248976624e2410f9d6ffd6ceb10b526 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7248976624e2410f9d6ffd6ceb10b526" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 45111 } }
I20240502 01:57:58.694604 7137 raft_consensus.cc:504] T a266cf6c4b8d4004808d091e48312078 P 00f7848b80b344e1968566450a055936 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00f7848b80b344e1968566450a055936" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 32875 } }
I20240502 01:57:58.694733 7137 leader_election.cc:302] T a266cf6c4b8d4004808d091e48312078 P 00f7848b80b344e1968566450a055936 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 00f7848b80b344e1968566450a055936; no voters:
I20240502 01:57:58.694605 7136 leader_election.cc:302] T 8263a9836e514c0d9c0504172816d961 P 7248976624e2410f9d6ffd6ceb10b526 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7248976624e2410f9d6ffd6ceb10b526; no voters:
I20240502 01:57:58.694893 7137 leader_election.cc:288] T a266cf6c4b8d4004808d091e48312078 P 00f7848b80b344e1968566450a055936 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:57:58.694991 7136 leader_election.cc:288] T 8263a9836e514c0d9c0504172816d961 P 7248976624e2410f9d6ffd6ceb10b526 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:57:58.695223 7137 ts_tablet_manager.cc:1398] T a266cf6c4b8d4004808d091e48312078 P 00f7848b80b344e1968566450a055936: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:57:58.695257 7136 ts_tablet_manager.cc:1398] T 8263a9836e514c0d9c0504172816d961 P 7248976624e2410f9d6ffd6ceb10b526: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:57:58.695883 6777 catalog_manager.cc:5484] T 3fec98c93c4f4fd6999c51b10f25d054 P 1266b98ba1bc4d8097ba226f05804bdc reported cstate change: term changed from 0 to 1, leader changed from <none> to 1266b98ba1bc4d8097ba226f05804bdc (127.6.117.194). New cstate: current_term: 1 leader_uuid: "1266b98ba1bc4d8097ba226f05804bdc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1266b98ba1bc4d8097ba226f05804bdc" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 39363 } health_report { overall_health: HEALTHY } } }
I20240502 01:57:58.696400 7137 tablet_bootstrap.cc:492] T c5ee2d565791431387fa0a0c5bc3a634 P 00f7848b80b344e1968566450a055936: Bootstrap starting.
I20240502 01:57:58.697193 7137 tablet_bootstrap.cc:654] T c5ee2d565791431387fa0a0c5bc3a634 P 00f7848b80b344e1968566450a055936: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:58.697666 7145 raft_consensus.cc:2781] T a266cf6c4b8d4004808d091e48312078 P 00f7848b80b344e1968566450a055936 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:57:58.697999 7145 raft_consensus.cc:686] T a266cf6c4b8d4004808d091e48312078 P 00f7848b80b344e1968566450a055936 [term 1 LEADER]: Becoming Leader. State: Replica: 00f7848b80b344e1968566450a055936, State: Running, Role: LEADER
I20240502 01:57:58.698130 7145 consensus_queue.cc:237] T a266cf6c4b8d4004808d091e48312078 P 00f7848b80b344e1968566450a055936 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00f7848b80b344e1968566450a055936" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 32875 } }
I20240502 01:57:58.699533 6774 catalog_manager.cc:5484] T a266cf6c4b8d4004808d091e48312078 P 00f7848b80b344e1968566450a055936 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00f7848b80b344e1968566450a055936 (127.6.117.193). New cstate: current_term: 1 leader_uuid: "00f7848b80b344e1968566450a055936" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00f7848b80b344e1968566450a055936" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 32875 } health_report { overall_health: HEALTHY } } }
I20240502 01:57:58.699977 7146 raft_consensus.cc:2781] T 8263a9836e514c0d9c0504172816d961 P 7248976624e2410f9d6ffd6ceb10b526 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:57:58.700449 7146 raft_consensus.cc:686] T 8263a9836e514c0d9c0504172816d961 P 7248976624e2410f9d6ffd6ceb10b526 [term 1 LEADER]: Becoming Leader. State: Replica: 7248976624e2410f9d6ffd6ceb10b526, State: Running, Role: LEADER
I20240502 01:57:58.702265 7146 consensus_queue.cc:237] T 8263a9836e514c0d9c0504172816d961 P 7248976624e2410f9d6ffd6ceb10b526 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7248976624e2410f9d6ffd6ceb10b526" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 45111 } }
I20240502 01:57:58.704658 7137 tablet_bootstrap.cc:492] T c5ee2d565791431387fa0a0c5bc3a634 P 00f7848b80b344e1968566450a055936: No bootstrap required, opened a new log
I20240502 01:57:58.704739 7137 ts_tablet_manager.cc:1367] T c5ee2d565791431387fa0a0c5bc3a634 P 00f7848b80b344e1968566450a055936: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240502 01:57:58.705386 7137 raft_consensus.cc:348] T c5ee2d565791431387fa0a0c5bc3a634 P 00f7848b80b344e1968566450a055936 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00f7848b80b344e1968566450a055936" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 32875 } }
I20240502 01:57:58.705464 7137 raft_consensus.cc:374] T c5ee2d565791431387fa0a0c5bc3a634 P 00f7848b80b344e1968566450a055936 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:58.705493 7137 raft_consensus.cc:729] T c5ee2d565791431387fa0a0c5bc3a634 P 00f7848b80b344e1968566450a055936 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00f7848b80b344e1968566450a055936, State: Initialized, Role: FOLLOWER
I20240502 01:57:58.705597 7137 consensus_queue.cc:260] T c5ee2d565791431387fa0a0c5bc3a634 P 00f7848b80b344e1968566450a055936 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00f7848b80b344e1968566450a055936" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 32875 } }
I20240502 01:57:58.705673 7137 raft_consensus.cc:388] T c5ee2d565791431387fa0a0c5bc3a634 P 00f7848b80b344e1968566450a055936 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:57:58.705706 7137 raft_consensus.cc:482] T c5ee2d565791431387fa0a0c5bc3a634 P 00f7848b80b344e1968566450a055936 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:57:58.705746 7137 raft_consensus.cc:3037] T c5ee2d565791431387fa0a0c5bc3a634 P 00f7848b80b344e1968566450a055936 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:58.706595 7137 raft_consensus.cc:504] T c5ee2d565791431387fa0a0c5bc3a634 P 00f7848b80b344e1968566450a055936 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00f7848b80b344e1968566450a055936" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 32875 } }
I20240502 01:57:58.706710 7137 leader_election.cc:302] T c5ee2d565791431387fa0a0c5bc3a634 P 00f7848b80b344e1968566450a055936 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 00f7848b80b344e1968566450a055936; no voters:
I20240502 01:57:58.706787 7137 leader_election.cc:288] T c5ee2d565791431387fa0a0c5bc3a634 P 00f7848b80b344e1968566450a055936 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:57:58.706985 7137 ts_tablet_manager.cc:1398] T c5ee2d565791431387fa0a0c5bc3a634 P 00f7848b80b344e1968566450a055936: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:57:58.707206 6774 catalog_manager.cc:5484] T 8263a9836e514c0d9c0504172816d961 P 7248976624e2410f9d6ffd6ceb10b526 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7248976624e2410f9d6ffd6ceb10b526 (127.6.117.195). New cstate: current_term: 1 leader_uuid: "7248976624e2410f9d6ffd6ceb10b526" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7248976624e2410f9d6ffd6ceb10b526" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 45111 } health_report { overall_health: HEALTHY } } }
I20240502 01:57:58.708971 7145 raft_consensus.cc:2781] T c5ee2d565791431387fa0a0c5bc3a634 P 00f7848b80b344e1968566450a055936 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:57:58.709045 7145 raft_consensus.cc:686] T c5ee2d565791431387fa0a0c5bc3a634 P 00f7848b80b344e1968566450a055936 [term 1 LEADER]: Becoming Leader. State: Replica: 00f7848b80b344e1968566450a055936, State: Running, Role: LEADER
I20240502 01:57:58.709136 7145 consensus_queue.cc:237] T c5ee2d565791431387fa0a0c5bc3a634 P 00f7848b80b344e1968566450a055936 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00f7848b80b344e1968566450a055936" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 32875 } }
I20240502 01:57:58.710359 6777 catalog_manager.cc:5484] T c5ee2d565791431387fa0a0c5bc3a634 P 00f7848b80b344e1968566450a055936 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00f7848b80b344e1968566450a055936 (127.6.117.193). New cstate: current_term: 1 leader_uuid: "00f7848b80b344e1968566450a055936" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00f7848b80b344e1968566450a055936" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 32875 } health_report { overall_health: HEALTHY } } }
I20240502 01:57:58.720978 6615 tablet_server.cc:178] TabletServer@127.6.117.193:0 shutting down...
I20240502 01:57:58.724815 6615 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:57:58.725062 6615 tablet_replica.cc:330] T c5ee2d565791431387fa0a0c5bc3a634 P 00f7848b80b344e1968566450a055936: stopping tablet replica
I20240502 01:57:58.725286 6615 raft_consensus.cc:2229] T c5ee2d565791431387fa0a0c5bc3a634 P 00f7848b80b344e1968566450a055936 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:57:58.725414 6615 pending_rounds.cc:62] T c5ee2d565791431387fa0a0c5bc3a634 P 00f7848b80b344e1968566450a055936: Trying to abort 1 pending ops.
I20240502 01:57:58.725463 6615 pending_rounds.cc:68] T c5ee2d565791431387fa0a0c5bc3a634 P 00f7848b80b344e1968566450a055936: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023063362392977408 op_type: NO_OP noop_request { }
I20240502 01:57:58.725540 6615 raft_consensus.cc:2866] T c5ee2d565791431387fa0a0c5bc3a634 P 00f7848b80b344e1968566450a055936 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240502 01:57:58.725597 6615 raft_consensus.cc:2258] T c5ee2d565791431387fa0a0c5bc3a634 P 00f7848b80b344e1968566450a055936 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240502 01:57:58.740629 7156 consensus_queue.cc:1159] T c5ee2d565791431387fa0a0c5bc3a634 P 00f7848b80b344e1968566450a055936 [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "00f7848b80b344e1968566450a055936" status { last_received { term: 1 index: 1 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 1 } }
I20240502 01:57:58.741807 6615 tablet_replica.cc:330] T a266cf6c4b8d4004808d091e48312078 P 00f7848b80b344e1968566450a055936: stopping tablet replica
I20240502 01:57:58.741925 6615 raft_consensus.cc:2229] T a266cf6c4b8d4004808d091e48312078 P 00f7848b80b344e1968566450a055936 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:57:58.742053 6615 raft_consensus.cc:2258] T a266cf6c4b8d4004808d091e48312078 P 00f7848b80b344e1968566450a055936 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:57:58.756076 6615 tablet_server.cc:194] TabletServer@127.6.117.193:0 shutdown complete.
I20240502 01:57:58.760035 6615 tablet_server.cc:178] TabletServer@127.6.117.194:0 shutting down...
I20240502 01:57:58.763402 6615 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:57:58.763665 6615 tablet_replica.cc:330] T 3fec98c93c4f4fd6999c51b10f25d054 P 1266b98ba1bc4d8097ba226f05804bdc: stopping tablet replica
I20240502 01:57:58.763792 6615 raft_consensus.cc:2229] T 3fec98c93c4f4fd6999c51b10f25d054 P 1266b98ba1bc4d8097ba226f05804bdc [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:57:58.763911 6615 raft_consensus.cc:2258] T 3fec98c93c4f4fd6999c51b10f25d054 P 1266b98ba1bc4d8097ba226f05804bdc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:57:58.776882 6615 tablet_server.cc:194] TabletServer@127.6.117.194:0 shutdown complete.
I20240502 01:57:58.778630 6615 tablet_server.cc:178] TabletServer@127.6.117.195:0 shutting down...
I20240502 01:57:58.782007 6615 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:57:58.782233 6615 tablet_replica.cc:330] T 8263a9836e514c0d9c0504172816d961 P 7248976624e2410f9d6ffd6ceb10b526: stopping tablet replica
I20240502 01:57:58.782342 6615 raft_consensus.cc:2229] T 8263a9836e514c0d9c0504172816d961 P 7248976624e2410f9d6ffd6ceb10b526 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:57:58.782447 6615 raft_consensus.cc:2258] T 8263a9836e514c0d9c0504172816d961 P 7248976624e2410f9d6ffd6ceb10b526 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:57:58.795070 6615 tablet_server.cc:194] TabletServer@127.6.117.195:0 shutdown complete.
I20240502 01:57:58.796972 6615 master.cc:537] Master@127.6.117.254:37651 shutting down...
I20240502 01:57:58.800248 6615 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 330ea26ef0a64042a8dbce0db3b4eee1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:57:58.800434 6615 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 330ea26ef0a64042a8dbce0db3b4eee1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:57:58.800506 6615 tablet_replica.cc:330] T 00000000000000000000000000000000 P 330ea26ef0a64042a8dbce0db3b4eee1: stopping tablet replica
I20240502 01:57:58.813316 6615 master.cc:552] Master@127.6.117.254:37651 shutdown complete.
I20240502 01:57:58.815840 6615 master.cc:537] Master@127.6.117.253:46275 shutting down...
I20240502 01:57:58.818619 6615 raft_consensus.cc:2229] T 00000000000000000000000000000000 P c9f6184fabb34268b6285bb1fa04a61d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:57:58.818764 6615 raft_consensus.cc:2258] T 00000000000000000000000000000000 P c9f6184fabb34268b6285bb1fa04a61d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:57:58.818825 6615 tablet_replica.cc:330] T 00000000000000000000000000000000 P c9f6184fabb34268b6285bb1fa04a61d: stopping tablet replica
I20240502 01:57:58.831468 6615 master.cc:552] Master@127.6.117.253:46275 shutdown complete.
I20240502 01:57:58.833899 6615 master.cc:537] Master@127.6.117.252:43201 shutting down...
I20240502 01:57:58.837173 6615 raft_consensus.cc:2229] T 00000000000000000000000000000000 P a0fea32478fa43999e85e0f0cefb9590 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:57:58.837463 6615 raft_consensus.cc:2258] T 00000000000000000000000000000000 P a0fea32478fa43999e85e0f0cefb9590 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:57:58.837527 6615 tablet_replica.cc:330] T 00000000000000000000000000000000 P a0fea32478fa43999e85e0f0cefb9590: stopping tablet replica
I20240502 01:57:58.850239 6615 master.cc:552] Master@127.6.117.252:43201 shutdown complete.
[ OK ] AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing (2658 ms)
[ RUN ] AutoRebalancerTest.AutoRebalancingTurnOffAndOn
I20240502 01:57:58.857574 6615 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.6.117.254:35435
I20240502 01:57:58.857893 6615 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:57:58.859375 7158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:57:58.859750 7159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:57:58.860052 7161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:57:58.860441 6615 server_base.cc:1034] running on GCE node
I20240502 01:57:58.860574 6615 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:57:58.860608 6615 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:57:58.860626 6615 hybrid_clock.cc:648] HybridClock initialized: now 1714615078860626 us; error 0 us; skew 500 ppm
I20240502 01:57:58.861495 6615 webserver.cc:460] Webserver started at http://127.6.117.254:32895/ using document root <none> and password file <none>
I20240502 01:57:58.861644 6615 fs_manager.cc:359] Metadata directory not provided
I20240502 01:57:58.861688 6615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:57:58.861758 6615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:57:58.862100 6615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714615076189642-6615-0/minicluster-data/master-0-root/instance:
uuid: "2bd74cf1295c44a1950e5eee7b5199fe"
format_stamp: "Formatted at 2024-05-02 01:57:58 on dist-test-slave-89cm"
I20240502 01:57:58.863416 6615 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240502 01:57:58.864193 7166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:58.864459 6615 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 01:57:58.864521 6615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714615076189642-6615-0/minicluster-data/master-0-root
uuid: "2bd74cf1295c44a1950e5eee7b5199fe"
format_stamp: "Formatted at 2024-05-02 01:57:58 on dist-test-slave-89cm"
I20240502 01:57:58.864584 6615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714615076189642-6615-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714615076189642-6615-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714615076189642-6615-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:57:58.874732 6615 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:57:58.875072 6615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:57:58.878973 6615 rpc_server.cc:307] RPC server started. Bound to: 127.6.117.254:35435
I20240502 01:57:58.881124 7218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.117.254:35435 every 8 connection(s)
I20240502 01:57:58.881516 7219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:58.883199 7219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bd74cf1295c44a1950e5eee7b5199fe: Bootstrap starting.
I20240502 01:57:58.883908 7219 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2bd74cf1295c44a1950e5eee7b5199fe: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:58.884861 7219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bd74cf1295c44a1950e5eee7b5199fe: No bootstrap required, opened a new log
I20240502 01:57:58.885185 7219 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2bd74cf1295c44a1950e5eee7b5199fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bd74cf1295c44a1950e5eee7b5199fe" member_type: VOTER }
I20240502 01:57:58.885267 7219 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2bd74cf1295c44a1950e5eee7b5199fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:58.885293 7219 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2bd74cf1295c44a1950e5eee7b5199fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bd74cf1295c44a1950e5eee7b5199fe, State: Initialized, Role: FOLLOWER
I20240502 01:57:58.885397 7219 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2bd74cf1295c44a1950e5eee7b5199fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bd74cf1295c44a1950e5eee7b5199fe" member_type: VOTER }
I20240502 01:57:58.885478 7219 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2bd74cf1295c44a1950e5eee7b5199fe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:57:58.885505 7219 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2bd74cf1295c44a1950e5eee7b5199fe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:57:58.885538 7219 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2bd74cf1295c44a1950e5eee7b5199fe [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:58.886133 7219 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2bd74cf1295c44a1950e5eee7b5199fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bd74cf1295c44a1950e5eee7b5199fe" member_type: VOTER }
I20240502 01:57:58.886240 7219 leader_election.cc:302] T 00000000000000000000000000000000 P 2bd74cf1295c44a1950e5eee7b5199fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2bd74cf1295c44a1950e5eee7b5199fe; no voters:
I20240502 01:57:58.886368 7219 leader_election.cc:288] T 00000000000000000000000000000000 P 2bd74cf1295c44a1950e5eee7b5199fe [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:57:58.886482 7222 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2bd74cf1295c44a1950e5eee7b5199fe [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:57:58.886662 7222 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2bd74cf1295c44a1950e5eee7b5199fe [term 1 LEADER]: Becoming Leader. State: Replica: 2bd74cf1295c44a1950e5eee7b5199fe, State: Running, Role: LEADER
I20240502 01:57:58.886741 7219 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2bd74cf1295c44a1950e5eee7b5199fe [sys.catalog]: configured and running, proceeding with master startup.
I20240502 01:57:58.886785 7222 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2bd74cf1295c44a1950e5eee7b5199fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bd74cf1295c44a1950e5eee7b5199fe" member_type: VOTER }
I20240502 01:57:58.887248 7223 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bd74cf1295c44a1950e5eee7b5199fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2bd74cf1295c44a1950e5eee7b5199fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bd74cf1295c44a1950e5eee7b5199fe" member_type: VOTER } }
I20240502 01:57:58.887270 7224 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bd74cf1295c44a1950e5eee7b5199fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2bd74cf1295c44a1950e5eee7b5199fe. Latest consensus state: current_term: 1 leader_uuid: "2bd74cf1295c44a1950e5eee7b5199fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bd74cf1295c44a1950e5eee7b5199fe" member_type: VOTER } }
I20240502 01:57:58.887354 7223 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bd74cf1295c44a1950e5eee7b5199fe [sys.catalog]: This master's current role is: LEADER
I20240502 01:57:58.887363 7224 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bd74cf1295c44a1950e5eee7b5199fe [sys.catalog]: This master's current role is: LEADER
I20240502 01:57:58.887601 7228 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 01:57:58.888464 7228 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 01:57:58.888695 6615 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 01:57:58.890139 7228 catalog_manager.cc:1347] Generated new cluster ID: f7d92d7846f44117b27d7945fcb607f8
I20240502 01:57:58.890185 7228 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 01:57:58.898212 7228 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 01:57:58.898762 7228 catalog_manager.cc:1504] Loading token signing keys...
I20240502 01:57:58.902499 7228 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 2bd74cf1295c44a1950e5eee7b5199fe: Generated new TSK 0
I20240502 01:57:58.902653 7228 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 01:57:58.904809 6615 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:57:58.906500 7241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:57:58.906849 7240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:57:58.907151 7243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:57:58.907626 6615 server_base.cc:1034] running on GCE node
I20240502 01:57:58.907789 6615 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:57:58.907825 6615 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:57:58.907840 6615 hybrid_clock.cc:648] HybridClock initialized: now 1714615078907839 us; error 0 us; skew 500 ppm
I20240502 01:57:58.908756 6615 webserver.cc:460] Webserver started at http://127.6.117.193:40357/ using document root <none> and password file <none>
I20240502 01:57:58.908898 6615 fs_manager.cc:359] Metadata directory not provided
I20240502 01:57:58.908946 6615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:57:58.909014 6615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:57:58.909371 6615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714615076189642-6615-0/minicluster-data/ts-0-root/instance:
uuid: "5279c463ab5f4b6aa1d50c3fd0536174"
format_stamp: "Formatted at 2024-05-02 01:57:58 on dist-test-slave-89cm"
I20240502 01:57:58.910673 6615 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240502 01:57:58.911473 7248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:58.911687 6615 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 01:57:58.911748 6615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714615076189642-6615-0/minicluster-data/ts-0-root
uuid: "5279c463ab5f4b6aa1d50c3fd0536174"
format_stamp: "Formatted at 2024-05-02 01:57:58 on dist-test-slave-89cm"
I20240502 01:57:58.911821 6615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714615076189642-6615-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714615076189642-6615-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714615076189642-6615-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:57:58.934542 6615 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:57:58.934940 6615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:57:58.935216 6615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:57:58.935688 6615 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:57:58.935727 6615 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:58.935770 6615 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:57:58.935804 6615 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:58.939613 6615 rpc_server.cc:307] RPC server started. Bound to: 127.6.117.193:44887
I20240502 01:57:58.939644 7311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.117.193:44887 every 8 connection(s)
I20240502 01:57:58.940418 6615 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:57:58.942188 7316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:57:58.942498 7317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:57:58.942819 7319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:57:58.943096 6615 server_base.cc:1034] running on GCE node
I20240502 01:57:58.943262 6615 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:57:58.943305 6615 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:57:58.943318 6615 hybrid_clock.cc:648] HybridClock initialized: now 1714615078943318 us; error 0 us; skew 500 ppm
I20240502 01:57:58.944231 6615 webserver.cc:460] Webserver started at http://127.6.117.194:35281/ using document root <none> and password file <none>
I20240502 01:57:58.944423 6615 fs_manager.cc:359] Metadata directory not provided
I20240502 01:57:58.944479 6615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:57:58.944541 6615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:57:58.944916 6615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714615076189642-6615-0/minicluster-data/ts-1-root/instance:
uuid: "97c52edf29a843bba2cd1653859f994e"
format_stamp: "Formatted at 2024-05-02 01:57:58 on dist-test-slave-89cm"
I20240502 01:57:58.945174 7312 heartbeater.cc:347] Connected to a master server at 127.6.117.254:35435
I20240502 01:57:58.945287 7312 heartbeater.cc:464] Registering TS with master...
I20240502 01:57:58.945514 7312 heartbeater.cc:511] Master 127.6.117.254:35435 requested a full tablet report, sending...
I20240502 01:57:58.946213 7183 ts_manager.cc:194] Registered new tserver with Master: 5279c463ab5f4b6aa1d50c3fd0536174 (127.6.117.193:44887)
I20240502 01:57:58.946509 6615 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240502 01:57:58.946991 7183 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56830
I20240502 01:57:58.947456 7324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:58.947695 6615 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240502 01:57:58.947752 6615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714615076189642-6615-0/minicluster-data/ts-1-root
uuid: "97c52edf29a843bba2cd1653859f994e"
format_stamp: "Formatted at 2024-05-02 01:57:58 on dist-test-slave-89cm"
I20240502 01:57:58.947830 6615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714615076189642-6615-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714615076189642-6615-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714615076189642-6615-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:57:58.959739 6615 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:57:58.960145 6615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:57:58.960471 6615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:57:58.960953 6615 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:57:58.960995 6615 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:58.961038 6615 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:57:58.961066 6615 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:58.964956 6615 rpc_server.cc:307] RPC server started. Bound to: 127.6.117.194:44123
I20240502 01:57:58.965005 7387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.117.194:44123 every 8 connection(s)
I20240502 01:57:58.965703 6615 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:57:58.967404 7391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:57:58.967689 7392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:57:58.968026 7394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:57:58.968326 6615 server_base.cc:1034] running on GCE node
I20240502 01:57:58.968528 6615 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:57:58.968581 6615 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:57:58.968602 6615 hybrid_clock.cc:648] HybridClock initialized: now 1714615078968602 us; error 0 us; skew 500 ppm
I20240502 01:57:58.969494 6615 webserver.cc:460] Webserver started at http://127.6.117.195:37627/ using document root <none> and password file <none>
I20240502 01:57:58.969642 6615 fs_manager.cc:359] Metadata directory not provided
I20240502 01:57:58.969699 6615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:57:58.969724 7388 heartbeater.cc:347] Connected to a master server at 127.6.117.254:35435
I20240502 01:57:58.969771 6615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:57:58.969820 7388 heartbeater.cc:464] Registering TS with master...
I20240502 01:57:58.970018 7388 heartbeater.cc:511] Master 127.6.117.254:35435 requested a full tablet report, sending...
I20240502 01:57:58.970224 6615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714615076189642-6615-0/minicluster-data/ts-2-root/instance:
uuid: "8b6a9ac9d6cd47b79f585d2981821b97"
format_stamp: "Formatted at 2024-05-02 01:57:58 on dist-test-slave-89cm"
I20240502 01:57:58.970636 7183 ts_manager.cc:194] Registered new tserver with Master: 97c52edf29a843bba2cd1653859f994e (127.6.117.194:44123)
I20240502 01:57:58.971299 7183 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56832
I20240502 01:57:58.971854 6615 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20240502 01:57:58.972713 7399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:58.972937 6615 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20240502 01:57:58.972995 6615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714615076189642-6615-0/minicluster-data/ts-2-root
uuid: "8b6a9ac9d6cd47b79f585d2981821b97"
format_stamp: "Formatted at 2024-05-02 01:57:58 on dist-test-slave-89cm"
I20240502 01:57:58.973059 6615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714615076189642-6615-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714615076189642-6615-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714615076189642-6615-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:57:58.984375 6615 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:57:58.984757 6615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:57:58.985046 6615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:57:58.985512 6615 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:57:58.985551 6615 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:58.985596 6615 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:57:58.985634 6615 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:58.989975 6615 rpc_server.cc:307] RPC server started. Bound to: 127.6.117.195:36729
I20240502 01:57:58.990046 7462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.117.195:36729 every 8 connection(s)
I20240502 01:57:58.994163 7463 heartbeater.cc:347] Connected to a master server at 127.6.117.254:35435
I20240502 01:57:58.994254 7463 heartbeater.cc:464] Registering TS with master...
I20240502 01:57:58.994446 7463 heartbeater.cc:511] Master 127.6.117.254:35435 requested a full tablet report, sending...
I20240502 01:57:58.995016 7183 ts_manager.cc:194] Registered new tserver with Master: 8b6a9ac9d6cd47b79f585d2981821b97 (127.6.117.195:36729)
I20240502 01:57:58.995501 7183 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56834
I20240502 01:57:58.995929 6615 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.005598031s
I20240502 01:57:59.948014 7312 heartbeater.cc:503] Master 127.6.117.254:35435 was elected leader, sending a full tablet report...
I20240502 01:57:59.972188 7388 heartbeater.cc:503] Master 127.6.117.254:35435 was elected leader, sending a full tablet report...
I20240502 01:57:59.996441 7463 heartbeater.cc:503] Master 127.6.117.254:35435 was elected leader, sending a full tablet report...
I20240502 01:58:00.022003 6615 test_util.cc:268] Using random seed: -289050641
I20240502 01:58:00.034440 7183 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56850:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\017\004\001\000\376\377\377\037\004\001\000\375\377\377/\004\001\000\374\377\377?\004\001\000\373\377\377O\004\001\000\372\377\377_\004\001\000\371\377\377o""\004\001\000\377\377\377\017\004\001\000\376\377\377\037\004\001\000\375\377\377/\004\001\000\374\377\377?\004\001\000\373\377\377O\004\001\000\372\377\377_\004\001\000\371\377\377o"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 01:58:00.035147 7183 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 01:58:00.048316 7276 tablet_service.cc:1467] Processing CreateTablet for tablet 01d518fafee34875a7df792d8297834b (DEFAULT_TABLE table=test-workload [id=f9d748f9e6e3463990f85ce3400747e9]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240502 01:58:00.048681 7276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01d518fafee34875a7df792d8297834b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:00.048728 7351 tablet_service.cc:1467] Processing CreateTablet for tablet ef99459b1bb145689ba08b01e28e4093 (DEFAULT_TABLE table=test-workload [id=f9d748f9e6e3463990f85ce3400747e9]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240502 01:58:00.049345 7269 tablet_service.cc:1467] Processing CreateTablet for tablet 7ba4aedf729d49bc977d5e52af888ebc (DEFAULT_TABLE table=test-workload [id=f9d748f9e6e3463990f85ce3400747e9]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240502 01:58:00.049371 7348 tablet_service.cc:1467] Processing CreateTablet for tablet e5668a318ea44db8a7c4db7fc7c95f7a (DEFAULT_TABLE table=test-workload [id=f9d748f9e6e3463990f85ce3400747e9]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240502 01:58:00.049579 7269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ba4aedf729d49bc977d5e52af888ebc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:00.049706 7347 tablet_service.cc:1467] Processing CreateTablet for tablet c1ce57f32584460f9bd94d85d19c8c4d (DEFAULT_TABLE table=test-workload [id=f9d748f9e6e3463990f85ce3400747e9]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240502 01:58:00.050029 7346 tablet_service.cc:1467] Processing CreateTablet for tablet f3e2a90036a74f5a81d94a49595dc62a (DEFAULT_TABLE table=test-workload [id=f9d748f9e6e3463990f85ce3400747e9]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240502 01:58:00.050336 7345 tablet_service.cc:1467] Processing CreateTablet for tablet 7ba4aedf729d49bc977d5e52af888ebc (DEFAULT_TABLE table=test-workload [id=f9d748f9e6e3463990f85ce3400747e9]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240502 01:58:00.050401 7427 tablet_service.cc:1467] Processing CreateTablet for tablet 01d518fafee34875a7df792d8297834b (DEFAULT_TABLE table=test-workload [id=f9d748f9e6e3463990f85ce3400747e9]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240502 01:58:00.050616 7427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01d518fafee34875a7df792d8297834b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:00.051043 7350 tablet_service.cc:1467] Processing CreateTablet for tablet 42d98987ce3e4e82ad0e55e00cd49c24 (DEFAULT_TABLE table=test-workload [id=f9d748f9e6e3463990f85ce3400747e9]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240502 01:58:00.051206 7423 tablet_service.cc:1467] Processing CreateTablet for tablet e5668a318ea44db8a7c4db7fc7c95f7a (DEFAULT_TABLE table=test-workload [id=f9d748f9e6e3463990f85ce3400747e9]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240502 01:58:00.051263 7349 tablet_service.cc:1467] Processing CreateTablet for tablet 7930116ca8d646faa009be917af87de8 (DEFAULT_TABLE table=test-workload [id=f9d748f9e6e3463990f85ce3400747e9]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240502 01:58:00.051429 7423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5668a318ea44db8a7c4db7fc7c95f7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:00.051704 7270 tablet_service.cc:1467] Processing CreateTablet for tablet f3e2a90036a74f5a81d94a49595dc62a (DEFAULT_TABLE table=test-workload [id=f9d748f9e6e3463990f85ce3400747e9]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240502 01:58:00.051928 7270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3e2a90036a74f5a81d94a49595dc62a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:00.051936 7351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef99459b1bb145689ba08b01e28e4093. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:00.053045 7426 tablet_service.cc:1467] Processing CreateTablet for tablet ef99459b1bb145689ba08b01e28e4093 (DEFAULT_TABLE table=test-workload [id=f9d748f9e6e3463990f85ce3400747e9]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240502 01:58:00.053344 7425 tablet_service.cc:1467] Processing CreateTablet for tablet 42d98987ce3e4e82ad0e55e00cd49c24 (DEFAULT_TABLE table=test-workload [id=f9d748f9e6e3463990f85ce3400747e9]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240502 01:58:00.053515 7426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef99459b1bb145689ba08b01e28e4093. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:00.053735 7424 tablet_service.cc:1467] Processing CreateTablet for tablet 7930116ca8d646faa009be917af87de8 (DEFAULT_TABLE table=test-workload [id=f9d748f9e6e3463990f85ce3400747e9]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240502 01:58:00.053886 7271 tablet_service.cc:1467] Processing CreateTablet for tablet c1ce57f32584460f9bd94d85d19c8c4d (DEFAULT_TABLE table=test-workload [id=f9d748f9e6e3463990f85ce3400747e9]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240502 01:58:00.053963 7424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7930116ca8d646faa009be917af87de8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:00.054105 7271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1ce57f32584460f9bd94d85d19c8c4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:00.054410 7346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3e2a90036a74f5a81d94a49595dc62a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:00.055212 7422 tablet_service.cc:1467] Processing CreateTablet for tablet c1ce57f32584460f9bd94d85d19c8c4d (DEFAULT_TABLE table=test-workload [id=f9d748f9e6e3463990f85ce3400747e9]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240502 01:58:00.055444 7422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1ce57f32584460f9bd94d85d19c8c4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:00.055722 7483 tablet_bootstrap.cc:492] T 01d518fafee34875a7df792d8297834b P 5279c463ab5f4b6aa1d50c3fd0536174: Bootstrap starting.
I20240502 01:58:00.056046 7272 tablet_service.cc:1467] Processing CreateTablet for tablet e5668a318ea44db8a7c4db7fc7c95f7a (DEFAULT_TABLE table=test-workload [id=f9d748f9e6e3463990f85ce3400747e9]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240502 01:58:00.056278 7272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5668a318ea44db8a7c4db7fc7c95f7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:00.056890 7483 tablet_bootstrap.cc:654] T 01d518fafee34875a7df792d8297834b P 5279c463ab5f4b6aa1d50c3fd0536174: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:00.056967 7348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5668a318ea44db8a7c4db7fc7c95f7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:00.057437 7421 tablet_service.cc:1467] Processing CreateTablet for tablet f3e2a90036a74f5a81d94a49595dc62a (DEFAULT_TABLE table=test-workload [id=f9d748f9e6e3463990f85ce3400747e9]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240502 01:58:00.057657 7421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3e2a90036a74f5a81d94a49595dc62a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:00.057955 7483 tablet_bootstrap.cc:492] T 01d518fafee34875a7df792d8297834b P 5279c463ab5f4b6aa1d50c3fd0536174: No bootstrap required, opened a new log
I20240502 01:58:00.058027 7483 ts_tablet_manager.cc:1367] T 01d518fafee34875a7df792d8297834b P 5279c463ab5f4b6aa1d50c3fd0536174: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:00.058179 7345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ba4aedf729d49bc977d5e52af888ebc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:00.058274 7273 tablet_service.cc:1467] Processing CreateTablet for tablet 7930116ca8d646faa009be917af87de8 (DEFAULT_TABLE table=test-workload [id=f9d748f9e6e3463990f85ce3400747e9]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240502 01:58:00.058501 7273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7930116ca8d646faa009be917af87de8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:00.058447 7483 raft_consensus.cc:348] T 01d518fafee34875a7df792d8297834b P 5279c463ab5f4b6aa1d50c3fd0536174 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } } peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } }
I20240502 01:58:00.058545 7483 raft_consensus.cc:374] T 01d518fafee34875a7df792d8297834b P 5279c463ab5f4b6aa1d50c3fd0536174 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:00.058574 7483 raft_consensus.cc:729] T 01d518fafee34875a7df792d8297834b P 5279c463ab5f4b6aa1d50c3fd0536174 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5279c463ab5f4b6aa1d50c3fd0536174, State: Initialized, Role: FOLLOWER
I20240502 01:58:00.058699 7483 consensus_queue.cc:260] T 01d518fafee34875a7df792d8297834b P 5279c463ab5f4b6aa1d50c3fd0536174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } } peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } }
I20240502 01:58:00.059053 7483 ts_tablet_manager.cc:1398] T 01d518fafee34875a7df792d8297834b P 5279c463ab5f4b6aa1d50c3fd0536174: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:00.059237 7483 tablet_bootstrap.cc:492] T 7ba4aedf729d49bc977d5e52af888ebc P 5279c463ab5f4b6aa1d50c3fd0536174: Bootstrap starting.
I20240502 01:58:00.059903 7349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7930116ca8d646faa009be917af87de8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:00.060072 7483 tablet_bootstrap.cc:654] T 7ba4aedf729d49bc977d5e52af888ebc P 5279c463ab5f4b6aa1d50c3fd0536174: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:00.060338 7350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42d98987ce3e4e82ad0e55e00cd49c24. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:00.060406 7420 tablet_service.cc:1467] Processing CreateTablet for tablet 7ba4aedf729d49bc977d5e52af888ebc (DEFAULT_TABLE table=test-workload [id=f9d748f9e6e3463990f85ce3400747e9]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240502 01:58:00.060478 7274 tablet_service.cc:1467] Processing CreateTablet for tablet 42d98987ce3e4e82ad0e55e00cd49c24 (DEFAULT_TABLE table=test-workload [id=f9d748f9e6e3463990f85ce3400747e9]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240502 01:58:00.060618 7420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ba4aedf729d49bc977d5e52af888ebc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:00.060703 7274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42d98987ce3e4e82ad0e55e00cd49c24. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:00.062407 7425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42d98987ce3e4e82ad0e55e00cd49c24. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:00.062460 7347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1ce57f32584460f9bd94d85d19c8c4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:00.062741 7275 tablet_service.cc:1467] Processing CreateTablet for tablet ef99459b1bb145689ba08b01e28e4093 (DEFAULT_TABLE table=test-workload [id=f9d748f9e6e3463990f85ce3400747e9]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240502 01:58:00.062973 7275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef99459b1bb145689ba08b01e28e4093. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:00.063099 7483 tablet_bootstrap.cc:492] T 7ba4aedf729d49bc977d5e52af888ebc P 5279c463ab5f4b6aa1d50c3fd0536174: No bootstrap required, opened a new log
I20240502 01:58:00.063169 7483 ts_tablet_manager.cc:1367] T 7ba4aedf729d49bc977d5e52af888ebc P 5279c463ab5f4b6aa1d50c3fd0536174: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20240502 01:58:00.063556 7483 raft_consensus.cc:348] T 7ba4aedf729d49bc977d5e52af888ebc P 5279c463ab5f4b6aa1d50c3fd0536174 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } }
I20240502 01:58:00.063652 7483 raft_consensus.cc:374] T 7ba4aedf729d49bc977d5e52af888ebc P 5279c463ab5f4b6aa1d50c3fd0536174 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:00.063679 7483 raft_consensus.cc:729] T 7ba4aedf729d49bc977d5e52af888ebc P 5279c463ab5f4b6aa1d50c3fd0536174 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5279c463ab5f4b6aa1d50c3fd0536174, State: Initialized, Role: FOLLOWER
I20240502 01:58:00.063793 7483 consensus_queue.cc:260] T 7ba4aedf729d49bc977d5e52af888ebc P 5279c463ab5f4b6aa1d50c3fd0536174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } }
I20240502 01:58:00.064141 7483 ts_tablet_manager.cc:1398] T 7ba4aedf729d49bc977d5e52af888ebc P 5279c463ab5f4b6aa1d50c3fd0536174: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:58:00.064327 7483 tablet_bootstrap.cc:492] T f3e2a90036a74f5a81d94a49595dc62a P 5279c463ab5f4b6aa1d50c3fd0536174: Bootstrap starting.
I20240502 01:58:00.064971 7352 tablet_service.cc:1467] Processing CreateTablet for tablet 01d518fafee34875a7df792d8297834b (DEFAULT_TABLE table=test-workload [id=f9d748f9e6e3463990f85ce3400747e9]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240502 01:58:00.065212 7483 tablet_bootstrap.cc:654] T f3e2a90036a74f5a81d94a49595dc62a P 5279c463ab5f4b6aa1d50c3fd0536174: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:00.065831 7352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01d518fafee34875a7df792d8297834b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:00.066603 7485 tablet_bootstrap.cc:492] T ef99459b1bb145689ba08b01e28e4093 P 97c52edf29a843bba2cd1653859f994e: Bootstrap starting.
I20240502 01:58:00.066989 7484 tablet_bootstrap.cc:492] T 01d518fafee34875a7df792d8297834b P 8b6a9ac9d6cd47b79f585d2981821b97: Bootstrap starting.
I20240502 01:58:00.067467 7485 tablet_bootstrap.cc:654] T ef99459b1bb145689ba08b01e28e4093 P 97c52edf29a843bba2cd1653859f994e: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:00.068425 7485 tablet_bootstrap.cc:492] T ef99459b1bb145689ba08b01e28e4093 P 97c52edf29a843bba2cd1653859f994e: No bootstrap required, opened a new log
I20240502 01:58:00.068528 7485 ts_tablet_manager.cc:1367] T ef99459b1bb145689ba08b01e28e4093 P 97c52edf29a843bba2cd1653859f994e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:00.068579 7484 tablet_bootstrap.cc:654] T 01d518fafee34875a7df792d8297834b P 8b6a9ac9d6cd47b79f585d2981821b97: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:00.068899 7483 tablet_bootstrap.cc:492] T f3e2a90036a74f5a81d94a49595dc62a P 5279c463ab5f4b6aa1d50c3fd0536174: No bootstrap required, opened a new log
I20240502 01:58:00.068949 7485 raft_consensus.cc:348] T ef99459b1bb145689ba08b01e28e4093 P 97c52edf29a843bba2cd1653859f994e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } }
I20240502 01:58:00.069031 7483 ts_tablet_manager.cc:1367] T f3e2a90036a74f5a81d94a49595dc62a P 5279c463ab5f4b6aa1d50c3fd0536174: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:58:00.069063 7485 raft_consensus.cc:374] T ef99459b1bb145689ba08b01e28e4093 P 97c52edf29a843bba2cd1653859f994e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:00.069213 7485 raft_consensus.cc:729] T ef99459b1bb145689ba08b01e28e4093 P 97c52edf29a843bba2cd1653859f994e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97c52edf29a843bba2cd1653859f994e, State: Initialized, Role: FOLLOWER
I20240502 01:58:00.069347 7485 consensus_queue.cc:260] T ef99459b1bb145689ba08b01e28e4093 P 97c52edf29a843bba2cd1653859f994e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } }
I20240502 01:58:00.069581 7484 tablet_bootstrap.cc:492] T 01d518fafee34875a7df792d8297834b P 8b6a9ac9d6cd47b79f585d2981821b97: No bootstrap required, opened a new log
I20240502 01:58:00.069664 7484 ts_tablet_manager.cc:1367] T 01d518fafee34875a7df792d8297834b P 8b6a9ac9d6cd47b79f585d2981821b97: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:58:00.069690 7485 ts_tablet_manager.cc:1398] T ef99459b1bb145689ba08b01e28e4093 P 97c52edf29a843bba2cd1653859f994e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:00.069808 7483 raft_consensus.cc:348] T f3e2a90036a74f5a81d94a49595dc62a P 5279c463ab5f4b6aa1d50c3fd0536174 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } }
I20240502 01:58:00.069962 7483 raft_consensus.cc:374] T f3e2a90036a74f5a81d94a49595dc62a P 5279c463ab5f4b6aa1d50c3fd0536174 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:00.070039 7483 raft_consensus.cc:729] T f3e2a90036a74f5a81d94a49595dc62a P 5279c463ab5f4b6aa1d50c3fd0536174 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5279c463ab5f4b6aa1d50c3fd0536174, State: Initialized, Role: FOLLOWER
I20240502 01:58:00.070046 7484 raft_consensus.cc:348] T 01d518fafee34875a7df792d8297834b P 8b6a9ac9d6cd47b79f585d2981821b97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } } peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } }
I20240502 01:58:00.070168 7484 raft_consensus.cc:374] T 01d518fafee34875a7df792d8297834b P 8b6a9ac9d6cd47b79f585d2981821b97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:00.070212 7484 raft_consensus.cc:729] T 01d518fafee34875a7df792d8297834b P 8b6a9ac9d6cd47b79f585d2981821b97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b6a9ac9d6cd47b79f585d2981821b97, State: Initialized, Role: FOLLOWER
I20240502 01:58:00.069878 7485 tablet_bootstrap.cc:492] T f3e2a90036a74f5a81d94a49595dc62a P 97c52edf29a843bba2cd1653859f994e: Bootstrap starting.
I20240502 01:58:00.070367 7484 consensus_queue.cc:260] T 01d518fafee34875a7df792d8297834b P 8b6a9ac9d6cd47b79f585d2981821b97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } } peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } }
I20240502 01:58:00.070397 7483 consensus_queue.cc:260] T f3e2a90036a74f5a81d94a49595dc62a P 5279c463ab5f4b6aa1d50c3fd0536174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } }
I20240502 01:58:00.070777 7483 ts_tablet_manager.cc:1398] T f3e2a90036a74f5a81d94a49595dc62a P 5279c463ab5f4b6aa1d50c3fd0536174: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:00.070986 7483 tablet_bootstrap.cc:492] T c1ce57f32584460f9bd94d85d19c8c4d P 5279c463ab5f4b6aa1d50c3fd0536174: Bootstrap starting.
I20240502 01:58:00.071153 7485 tablet_bootstrap.cc:654] T f3e2a90036a74f5a81d94a49595dc62a P 97c52edf29a843bba2cd1653859f994e: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:00.071403 7484 ts_tablet_manager.cc:1398] T 01d518fafee34875a7df792d8297834b P 8b6a9ac9d6cd47b79f585d2981821b97: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:00.071951 7484 tablet_bootstrap.cc:492] T e5668a318ea44db8a7c4db7fc7c95f7a P 8b6a9ac9d6cd47b79f585d2981821b97: Bootstrap starting.
I20240502 01:58:00.071971 7483 tablet_bootstrap.cc:654] T c1ce57f32584460f9bd94d85d19c8c4d P 5279c463ab5f4b6aa1d50c3fd0536174: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:00.072183 7485 tablet_bootstrap.cc:492] T f3e2a90036a74f5a81d94a49595dc62a P 97c52edf29a843bba2cd1653859f994e: No bootstrap required, opened a new log
I20240502 01:58:00.072577 7485 ts_tablet_manager.cc:1367] T f3e2a90036a74f5a81d94a49595dc62a P 97c52edf29a843bba2cd1653859f994e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:58:00.072794 7484 tablet_bootstrap.cc:654] T e5668a318ea44db8a7c4db7fc7c95f7a P 8b6a9ac9d6cd47b79f585d2981821b97: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:00.072984 7485 raft_consensus.cc:348] T f3e2a90036a74f5a81d94a49595dc62a P 97c52edf29a843bba2cd1653859f994e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } }
I20240502 01:58:00.073083 7485 raft_consensus.cc:374] T f3e2a90036a74f5a81d94a49595dc62a P 97c52edf29a843bba2cd1653859f994e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:00.073113 7485 raft_consensus.cc:729] T f3e2a90036a74f5a81d94a49595dc62a P 97c52edf29a843bba2cd1653859f994e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97c52edf29a843bba2cd1653859f994e, State: Initialized, Role: FOLLOWER
I20240502 01:58:00.073248 7485 consensus_queue.cc:260] T f3e2a90036a74f5a81d94a49595dc62a P 97c52edf29a843bba2cd1653859f994e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } }
I20240502 01:58:00.073596 7485 ts_tablet_manager.cc:1398] T f3e2a90036a74f5a81d94a49595dc62a P 97c52edf29a843bba2cd1653859f994e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:00.073745 7485 tablet_bootstrap.cc:492] T 7ba4aedf729d49bc977d5e52af888ebc P 97c52edf29a843bba2cd1653859f994e: Bootstrap starting.
I20240502 01:58:00.074069 7484 tablet_bootstrap.cc:492] T e5668a318ea44db8a7c4db7fc7c95f7a P 8b6a9ac9d6cd47b79f585d2981821b97: No bootstrap required, opened a new log
I20240502 01:58:00.074142 7484 ts_tablet_manager.cc:1367] T e5668a318ea44db8a7c4db7fc7c95f7a P 8b6a9ac9d6cd47b79f585d2981821b97: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:00.074532 7485 tablet_bootstrap.cc:654] T 7ba4aedf729d49bc977d5e52af888ebc P 97c52edf29a843bba2cd1653859f994e: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:00.074492 7484 raft_consensus.cc:348] T e5668a318ea44db8a7c4db7fc7c95f7a P 8b6a9ac9d6cd47b79f585d2981821b97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } }
I20240502 01:58:00.074579 7484 raft_consensus.cc:374] T e5668a318ea44db8a7c4db7fc7c95f7a P 8b6a9ac9d6cd47b79f585d2981821b97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:00.074605 7484 raft_consensus.cc:729] T e5668a318ea44db8a7c4db7fc7c95f7a P 8b6a9ac9d6cd47b79f585d2981821b97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b6a9ac9d6cd47b79f585d2981821b97, State: Initialized, Role: FOLLOWER
I20240502 01:58:00.074702 7484 consensus_queue.cc:260] T e5668a318ea44db8a7c4db7fc7c95f7a P 8b6a9ac9d6cd47b79f585d2981821b97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } }
I20240502 01:58:00.074934 7484 ts_tablet_manager.cc:1398] T e5668a318ea44db8a7c4db7fc7c95f7a P 8b6a9ac9d6cd47b79f585d2981821b97: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:00.075104 7484 tablet_bootstrap.cc:492] T ef99459b1bb145689ba08b01e28e4093 P 8b6a9ac9d6cd47b79f585d2981821b97: Bootstrap starting.
I20240502 01:58:00.075881 7484 tablet_bootstrap.cc:654] T ef99459b1bb145689ba08b01e28e4093 P 8b6a9ac9d6cd47b79f585d2981821b97: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:00.076479 7494 raft_consensus.cc:482] T 01d518fafee34875a7df792d8297834b P 8b6a9ac9d6cd47b79f585d2981821b97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:00.076565 7494 raft_consensus.cc:504] T 01d518fafee34875a7df792d8297834b P 8b6a9ac9d6cd47b79f585d2981821b97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } } peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } }
I20240502 01:58:00.076768 7484 tablet_bootstrap.cc:492] T ef99459b1bb145689ba08b01e28e4093 P 8b6a9ac9d6cd47b79f585d2981821b97: No bootstrap required, opened a new log
I20240502 01:58:00.076848 7484 ts_tablet_manager.cc:1367] T ef99459b1bb145689ba08b01e28e4093 P 8b6a9ac9d6cd47b79f585d2981821b97: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:58:00.077039 7494 leader_election.cc:288] T 01d518fafee34875a7df792d8297834b P 8b6a9ac9d6cd47b79f585d2981821b97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5279c463ab5f4b6aa1d50c3fd0536174 (127.6.117.193:44887), 97c52edf29a843bba2cd1653859f994e (127.6.117.194:44123)
I20240502 01:58:00.077260 7484 raft_consensus.cc:348] T ef99459b1bb145689ba08b01e28e4093 P 8b6a9ac9d6cd47b79f585d2981821b97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } }
I20240502 01:58:00.077351 7484 raft_consensus.cc:374] T ef99459b1bb145689ba08b01e28e4093 P 8b6a9ac9d6cd47b79f585d2981821b97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:00.077383 7484 raft_consensus.cc:729] T ef99459b1bb145689ba08b01e28e4093 P 8b6a9ac9d6cd47b79f585d2981821b97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b6a9ac9d6cd47b79f585d2981821b97, State: Initialized, Role: FOLLOWER
I20240502 01:58:00.077492 7484 consensus_queue.cc:260] T ef99459b1bb145689ba08b01e28e4093 P 8b6a9ac9d6cd47b79f585d2981821b97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } }
I20240502 01:58:00.077921 7484 ts_tablet_manager.cc:1398] T ef99459b1bb145689ba08b01e28e4093 P 8b6a9ac9d6cd47b79f585d2981821b97: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:58:00.078096 7484 tablet_bootstrap.cc:492] T c1ce57f32584460f9bd94d85d19c8c4d P 8b6a9ac9d6cd47b79f585d2981821b97: Bootstrap starting.
I20240502 01:58:00.078907 7484 tablet_bootstrap.cc:654] T c1ce57f32584460f9bd94d85d19c8c4d P 8b6a9ac9d6cd47b79f585d2981821b97: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:00.079739 7483 tablet_bootstrap.cc:492] T c1ce57f32584460f9bd94d85d19c8c4d P 5279c463ab5f4b6aa1d50c3fd0536174: No bootstrap required, opened a new log
I20240502 01:58:00.079816 7483 ts_tablet_manager.cc:1367] T c1ce57f32584460f9bd94d85d19c8c4d P 5279c463ab5f4b6aa1d50c3fd0536174: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240502 01:58:00.080145 7483 raft_consensus.cc:348] T c1ce57f32584460f9bd94d85d19c8c4d P 5279c463ab5f4b6aa1d50c3fd0536174 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } }
I20240502 01:58:00.080242 7483 raft_consensus.cc:374] T c1ce57f32584460f9bd94d85d19c8c4d P 5279c463ab5f4b6aa1d50c3fd0536174 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:00.080322 7483 raft_consensus.cc:729] T c1ce57f32584460f9bd94d85d19c8c4d P 5279c463ab5f4b6aa1d50c3fd0536174 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5279c463ab5f4b6aa1d50c3fd0536174, State: Initialized, Role: FOLLOWER
I20240502 01:58:00.080449 7483 consensus_queue.cc:260] T c1ce57f32584460f9bd94d85d19c8c4d P 5279c463ab5f4b6aa1d50c3fd0536174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } }
I20240502 01:58:00.080679 7483 ts_tablet_manager.cc:1398] T c1ce57f32584460f9bd94d85d19c8c4d P 5279c463ab5f4b6aa1d50c3fd0536174: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:00.080842 7483 tablet_bootstrap.cc:492] T e5668a318ea44db8a7c4db7fc7c95f7a P 5279c463ab5f4b6aa1d50c3fd0536174: Bootstrap starting.
I20240502 01:58:00.081359 7286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01d518fafee34875a7df792d8297834b" candidate_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" is_pre_election: true
I20240502 01:58:00.081480 7286 raft_consensus.cc:2454] T 01d518fafee34875a7df792d8297834b P 5279c463ab5f4b6aa1d50c3fd0536174 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b6a9ac9d6cd47b79f585d2981821b97 in term 0.
I20240502 01:58:00.081648 7483 tablet_bootstrap.cc:654] T e5668a318ea44db8a7c4db7fc7c95f7a P 5279c463ab5f4b6aa1d50c3fd0536174: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:00.081898 7400 leader_election.cc:302] T 01d518fafee34875a7df792d8297834b P 8b6a9ac9d6cd47b79f585d2981821b97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5279c463ab5f4b6aa1d50c3fd0536174, 8b6a9ac9d6cd47b79f585d2981821b97; no voters:
I20240502 01:58:00.082191 7494 raft_consensus.cc:2781] T 01d518fafee34875a7df792d8297834b P 8b6a9ac9d6cd47b79f585d2981821b97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:00.082257 7494 raft_consensus.cc:482] T 01d518fafee34875a7df792d8297834b P 8b6a9ac9d6cd47b79f585d2981821b97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:00.082305 7494 raft_consensus.cc:3037] T 01d518fafee34875a7df792d8297834b P 8b6a9ac9d6cd47b79f585d2981821b97 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:00.082682 7484 tablet_bootstrap.cc:492] T c1ce57f32584460f9bd94d85d19c8c4d P 8b6a9ac9d6cd47b79f585d2981821b97: No bootstrap required, opened a new log
I20240502 01:58:00.082757 7484 ts_tablet_manager.cc:1367] T c1ce57f32584460f9bd94d85d19c8c4d P 8b6a9ac9d6cd47b79f585d2981821b97: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20240502 01:58:00.083070 7484 raft_consensus.cc:348] T c1ce57f32584460f9bd94d85d19c8c4d P 8b6a9ac9d6cd47b79f585d2981821b97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } }
I20240502 01:58:00.083150 7484 raft_consensus.cc:374] T c1ce57f32584460f9bd94d85d19c8c4d P 8b6a9ac9d6cd47b79f585d2981821b97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:00.083178 7484 raft_consensus.cc:729] T c1ce57f32584460f9bd94d85d19c8c4d P 8b6a9ac9d6cd47b79f585d2981821b97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b6a9ac9d6cd47b79f585d2981821b97, State: Initialized, Role: FOLLOWER
I20240502 01:58:00.083288 7484 consensus_queue.cc:260] T c1ce57f32584460f9bd94d85d19c8c4d P 8b6a9ac9d6cd47b79f585d2981821b97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } }
I20240502 01:58:00.083633 7484 ts_tablet_manager.cc:1398] T c1ce57f32584460f9bd94d85d19c8c4d P 8b6a9ac9d6cd47b79f585d2981821b97: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:00.083802 7484 tablet_bootstrap.cc:492] T f3e2a90036a74f5a81d94a49595dc62a P 8b6a9ac9d6cd47b79f585d2981821b97: Bootstrap starting.
I20240502 01:58:00.084450 7494 raft_consensus.cc:504] T 01d518fafee34875a7df792d8297834b P 8b6a9ac9d6cd47b79f585d2981821b97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } } peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } }
I20240502 01:58:00.084779 7494 leader_election.cc:288] T 01d518fafee34875a7df792d8297834b P 8b6a9ac9d6cd47b79f585d2981821b97 [CANDIDATE]: Term 1 election: Requested vote from peers 5279c463ab5f4b6aa1d50c3fd0536174 (127.6.117.193:44887), 97c52edf29a843bba2cd1653859f994e (127.6.117.194:44123)
I20240502 01:58:00.084640 7484 tablet_bootstrap.cc:654] T f3e2a90036a74f5a81d94a49595dc62a P 8b6a9ac9d6cd47b79f585d2981821b97: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:00.084918 7362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01d518fafee34875a7df792d8297834b" candidate_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97c52edf29a843bba2cd1653859f994e" is_pre_election: true
I20240502 01:58:00.085100 7361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01d518fafee34875a7df792d8297834b" candidate_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97c52edf29a843bba2cd1653859f994e"
I20240502 01:58:00.085302 7286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01d518fafee34875a7df792d8297834b" candidate_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5279c463ab5f4b6aa1d50c3fd0536174"
I20240502 01:58:00.085388 7286 raft_consensus.cc:3037] T 01d518fafee34875a7df792d8297834b P 5279c463ab5f4b6aa1d50c3fd0536174 [term 0 FOLLOWER]: Advancing to term 1
W20240502 01:58:00.085425 7400 leader_election.cc:341] T 01d518fafee34875a7df792d8297834b P 8b6a9ac9d6cd47b79f585d2981821b97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 97c52edf29a843bba2cd1653859f994e (127.6.117.194:44123): Illegal state: must be running to vote when last-logged opid is not known
W20240502 01:58:00.085569 7400 leader_election.cc:341] T 01d518fafee34875a7df792d8297834b P 8b6a9ac9d6cd47b79f585d2981821b97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97c52edf29a843bba2cd1653859f994e (127.6.117.194:44123): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:58:00.085734 7483 tablet_bootstrap.cc:492] T e5668a318ea44db8a7c4db7fc7c95f7a P 5279c463ab5f4b6aa1d50c3fd0536174: No bootstrap required, opened a new log
I20240502 01:58:00.085803 7483 ts_tablet_manager.cc:1367] T e5668a318ea44db8a7c4db7fc7c95f7a P 5279c463ab5f4b6aa1d50c3fd0536174: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240502 01:58:00.086148 7483 raft_consensus.cc:348] T e5668a318ea44db8a7c4db7fc7c95f7a P 5279c463ab5f4b6aa1d50c3fd0536174 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } }
I20240502 01:58:00.086230 7483 raft_consensus.cc:374] T e5668a318ea44db8a7c4db7fc7c95f7a P 5279c463ab5f4b6aa1d50c3fd0536174 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:00.086258 7483 raft_consensus.cc:729] T e5668a318ea44db8a7c4db7fc7c95f7a P 5279c463ab5f4b6aa1d50c3fd0536174 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5279c463ab5f4b6aa1d50c3fd0536174, State: Initialized, Role: FOLLOWER
I20240502 01:58:00.086369 7483 consensus_queue.cc:260] T e5668a318ea44db8a7c4db7fc7c95f7a P 5279c463ab5f4b6aa1d50c3fd0536174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } }
I20240502 01:58:00.086654 7483 ts_tablet_manager.cc:1398] T e5668a318ea44db8a7c4db7fc7c95f7a P 5279c463ab5f4b6aa1d50c3fd0536174: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:00.086863 7483 tablet_bootstrap.cc:492] T 7930116ca8d646faa009be917af87de8 P 5279c463ab5f4b6aa1d50c3fd0536174: Bootstrap starting.
I20240502 01:58:00.087740 7483 tablet_bootstrap.cc:654] T 7930116ca8d646faa009be917af87de8 P 5279c463ab5f4b6aa1d50c3fd0536174: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:00.088694 7484 tablet_bootstrap.cc:492] T f3e2a90036a74f5a81d94a49595dc62a P 8b6a9ac9d6cd47b79f585d2981821b97: No bootstrap required, opened a new log
I20240502 01:58:00.088780 7484 ts_tablet_manager.cc:1367] T f3e2a90036a74f5a81d94a49595dc62a P 8b6a9ac9d6cd47b79f585d2981821b97: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20240502 01:58:00.088816 7485 tablet_bootstrap.cc:492] T 7ba4aedf729d49bc977d5e52af888ebc P 97c52edf29a843bba2cd1653859f994e: No bootstrap required, opened a new log
I20240502 01:58:00.088883 7485 ts_tablet_manager.cc:1367] T 7ba4aedf729d49bc977d5e52af888ebc P 97c52edf29a843bba2cd1653859f994e: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240502 01:58:00.089145 7484 raft_consensus.cc:348] T f3e2a90036a74f5a81d94a49595dc62a P 8b6a9ac9d6cd47b79f585d2981821b97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } }
I20240502 01:58:00.089253 7484 raft_consensus.cc:374] T f3e2a90036a74f5a81d94a49595dc62a P 8b6a9ac9d6cd47b79f585d2981821b97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:00.089254 7485 raft_consensus.cc:348] T 7ba4aedf729d49bc977d5e52af888ebc P 97c52edf29a843bba2cd1653859f994e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } }
I20240502 01:58:00.089339 7484 raft_consensus.cc:729] T f3e2a90036a74f5a81d94a49595dc62a P 8b6a9ac9d6cd47b79f585d2981821b97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b6a9ac9d6cd47b79f585d2981821b97, State: Initialized, Role: FOLLOWER
I20240502 01:58:00.089357 7485 raft_consensus.cc:374] T 7ba4aedf729d49bc977d5e52af888ebc P 97c52edf29a843bba2cd1653859f994e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:00.089395 7485 raft_consensus.cc:729] T 7ba4aedf729d49bc977d5e52af888ebc P 97c52edf29a843bba2cd1653859f994e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97c52edf29a843bba2cd1653859f994e, State: Initialized, Role: FOLLOWER
I20240502 01:58:00.089449 7484 consensus_queue.cc:260] T f3e2a90036a74f5a81d94a49595dc62a P 8b6a9ac9d6cd47b79f585d2981821b97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } }
I20240502 01:58:00.089493 7485 consensus_queue.cc:260] T 7ba4aedf729d49bc977d5e52af888ebc P 97c52edf29a843bba2cd1653859f994e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } }
I20240502 01:58:00.089700 7484 ts_tablet_manager.cc:1398] T f3e2a90036a74f5a81d94a49595dc62a P 8b6a9ac9d6cd47b79f585d2981821b97: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:58:00.089725 7485 ts_tablet_manager.cc:1398] T 7ba4aedf729d49bc977d5e52af888ebc P 97c52edf29a843bba2cd1653859f994e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:00.089857 7485 tablet_bootstrap.cc:492] T 42d98987ce3e4e82ad0e55e00cd49c24 P 97c52edf29a843bba2cd1653859f994e: Bootstrap starting.
I20240502 01:58:00.089857 7484 tablet_bootstrap.cc:492] T 7ba4aedf729d49bc977d5e52af888ebc P 8b6a9ac9d6cd47b79f585d2981821b97: Bootstrap starting.
I20240502 01:58:00.090615 7483 tablet_bootstrap.cc:492] T 7930116ca8d646faa009be917af87de8 P 5279c463ab5f4b6aa1d50c3fd0536174: No bootstrap required, opened a new log
I20240502 01:58:00.090639 7484 tablet_bootstrap.cc:654] T 7ba4aedf729d49bc977d5e52af888ebc P 8b6a9ac9d6cd47b79f585d2981821b97: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:00.090689 7483 ts_tablet_manager.cc:1367] T 7930116ca8d646faa009be917af87de8 P 5279c463ab5f4b6aa1d50c3fd0536174: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240502 01:58:00.090765 7485 tablet_bootstrap.cc:654] T 42d98987ce3e4e82ad0e55e00cd49c24 P 97c52edf29a843bba2cd1653859f994e: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:00.091063 7483 raft_consensus.cc:348] T 7930116ca8d646faa009be917af87de8 P 5279c463ab5f4b6aa1d50c3fd0536174 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } }
I20240502 01:58:00.091150 7483 raft_consensus.cc:374] T 7930116ca8d646faa009be917af87de8 P 5279c463ab5f4b6aa1d50c3fd0536174 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:00.091189 7483 raft_consensus.cc:729] T 7930116ca8d646faa009be917af87de8 P 5279c463ab5f4b6aa1d50c3fd0536174 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5279c463ab5f4b6aa1d50c3fd0536174, State: Initialized, Role: FOLLOWER
I20240502 01:58:00.091308 7483 consensus_queue.cc:260] T 7930116ca8d646faa009be917af87de8 P 5279c463ab5f4b6aa1d50c3fd0536174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } }
I20240502 01:58:00.091569 7483 ts_tablet_manager.cc:1398] T 7930116ca8d646faa009be917af87de8 P 5279c463ab5f4b6aa1d50c3fd0536174: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:00.092015 7484 tablet_bootstrap.cc:492] T 7ba4aedf729d49bc977d5e52af888ebc P 8b6a9ac9d6cd47b79f585d2981821b97: No bootstrap required, opened a new log
I20240502 01:58:00.092098 7484 ts_tablet_manager.cc:1367] T 7ba4aedf729d49bc977d5e52af888ebc P 8b6a9ac9d6cd47b79f585d2981821b97: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:58:00.092396 7485 tablet_bootstrap.cc:492] T 42d98987ce3e4e82ad0e55e00cd49c24 P 97c52edf29a843bba2cd1653859f994e: No bootstrap required, opened a new log
I20240502 01:58:00.092485 7485 ts_tablet_manager.cc:1367] T 42d98987ce3e4e82ad0e55e00cd49c24 P 97c52edf29a843bba2cd1653859f994e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:58:00.092788 7484 raft_consensus.cc:348] T 7ba4aedf729d49bc977d5e52af888ebc P 8b6a9ac9d6cd47b79f585d2981821b97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } }
I20240502 01:58:00.092892 7484 raft_consensus.cc:374] T 7ba4aedf729d49bc977d5e52af888ebc P 8b6a9ac9d6cd47b79f585d2981821b97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:00.092962 7484 raft_consensus.cc:729] T 7ba4aedf729d49bc977d5e52af888ebc P 8b6a9ac9d6cd47b79f585d2981821b97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b6a9ac9d6cd47b79f585d2981821b97, State: Initialized, Role: FOLLOWER
I20240502 01:58:00.093082 7484 consensus_queue.cc:260] T 7ba4aedf729d49bc977d5e52af888ebc P 8b6a9ac9d6cd47b79f585d2981821b97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } }
I20240502 01:58:00.093478 7484 ts_tablet_manager.cc:1398] T 7ba4aedf729d49bc977d5e52af888ebc P 8b6a9ac9d6cd47b79f585d2981821b97: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:00.092414 7483 tablet_bootstrap.cc:492] T 42d98987ce3e4e82ad0e55e00cd49c24 P 5279c463ab5f4b6aa1d50c3fd0536174: Bootstrap starting.
I20240502 01:58:00.093544 7286 raft_consensus.cc:2454] T 01d518fafee34875a7df792d8297834b P 5279c463ab5f4b6aa1d50c3fd0536174 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b6a9ac9d6cd47b79f585d2981821b97 in term 1.
I20240502 01:58:00.092877 7485 raft_consensus.cc:348] T 42d98987ce3e4e82ad0e55e00cd49c24 P 97c52edf29a843bba2cd1653859f994e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } }
I20240502 01:58:00.093725 7485 raft_consensus.cc:374] T 42d98987ce3e4e82ad0e55e00cd49c24 P 97c52edf29a843bba2cd1653859f994e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:00.093763 7485 raft_consensus.cc:729] T 42d98987ce3e4e82ad0e55e00cd49c24 P 97c52edf29a843bba2cd1653859f994e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97c52edf29a843bba2cd1653859f994e, State: Initialized, Role: FOLLOWER
I20240502 01:58:00.093863 7485 consensus_queue.cc:260] T 42d98987ce3e4e82ad0e55e00cd49c24 P 97c52edf29a843bba2cd1653859f994e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } }
I20240502 01:58:00.094059 7400 leader_election.cc:302] T 01d518fafee34875a7df792d8297834b P 8b6a9ac9d6cd47b79f585d2981821b97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5279c463ab5f4b6aa1d50c3fd0536174, 8b6a9ac9d6cd47b79f585d2981821b97; no voters: 97c52edf29a843bba2cd1653859f994e
I20240502 01:58:00.094126 7485 ts_tablet_manager.cc:1398] T 42d98987ce3e4e82ad0e55e00cd49c24 P 97c52edf29a843bba2cd1653859f994e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:00.094292 7485 tablet_bootstrap.cc:492] T c1ce57f32584460f9bd94d85d19c8c4d P 97c52edf29a843bba2cd1653859f994e: Bootstrap starting.
I20240502 01:58:00.094332 7505 raft_consensus.cc:2781] T 01d518fafee34875a7df792d8297834b P 8b6a9ac9d6cd47b79f585d2981821b97 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:00.094399 7505 raft_consensus.cc:686] T 01d518fafee34875a7df792d8297834b P 8b6a9ac9d6cd47b79f585d2981821b97 [term 1 LEADER]: Becoming Leader. State: Replica: 8b6a9ac9d6cd47b79f585d2981821b97, State: Running, Role: LEADER
I20240502 01:58:00.094508 7505 consensus_queue.cc:237] T 01d518fafee34875a7df792d8297834b P 8b6a9ac9d6cd47b79f585d2981821b97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } } peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } }
I20240502 01:58:00.095024 7485 tablet_bootstrap.cc:654] T c1ce57f32584460f9bd94d85d19c8c4d P 97c52edf29a843bba2cd1653859f994e: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:00.095206 7484 tablet_bootstrap.cc:492] T 42d98987ce3e4e82ad0e55e00cd49c24 P 8b6a9ac9d6cd47b79f585d2981821b97: Bootstrap starting.
I20240502 01:58:00.095943 7485 tablet_bootstrap.cc:492] T c1ce57f32584460f9bd94d85d19c8c4d P 97c52edf29a843bba2cd1653859f994e: No bootstrap required, opened a new log
I20240502 01:58:00.096617 7484 tablet_bootstrap.cc:654] T 42d98987ce3e4e82ad0e55e00cd49c24 P 8b6a9ac9d6cd47b79f585d2981821b97: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:00.097240 7485 ts_tablet_manager.cc:1367] T c1ce57f32584460f9bd94d85d19c8c4d P 97c52edf29a843bba2cd1653859f994e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:58:00.097618 7485 raft_consensus.cc:348] T c1ce57f32584460f9bd94d85d19c8c4d P 97c52edf29a843bba2cd1653859f994e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } }
I20240502 01:58:00.097714 7485 raft_consensus.cc:374] T c1ce57f32584460f9bd94d85d19c8c4d P 97c52edf29a843bba2cd1653859f994e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:00.097746 7485 raft_consensus.cc:729] T c1ce57f32584460f9bd94d85d19c8c4d P 97c52edf29a843bba2cd1653859f994e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97c52edf29a843bba2cd1653859f994e, State: Initialized, Role: FOLLOWER
I20240502 01:58:00.097651 7182 catalog_manager.cc:5484] T 01d518fafee34875a7df792d8297834b P 8b6a9ac9d6cd47b79f585d2981821b97 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b6a9ac9d6cd47b79f585d2981821b97 (127.6.117.195). New cstate: current_term: 1 leader_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:00.097843 7485 consensus_queue.cc:260] T c1ce57f32584460f9bd94d85d19c8c4d P 97c52edf29a843bba2cd1653859f994e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } }
I20240502 01:58:00.098084 7485 ts_tablet_manager.cc:1398] T c1ce57f32584460f9bd94d85d19c8c4d P 97c52edf29a843bba2cd1653859f994e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:00.097128 7483 tablet_bootstrap.cc:654] T 42d98987ce3e4e82ad0e55e00cd49c24 P 5279c463ab5f4b6aa1d50c3fd0536174: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:00.098743 7485 tablet_bootstrap.cc:492] T 7930116ca8d646faa009be917af87de8 P 97c52edf29a843bba2cd1653859f994e: Bootstrap starting.
I20240502 01:58:00.098762 7484 tablet_bootstrap.cc:492] T 42d98987ce3e4e82ad0e55e00cd49c24 P 8b6a9ac9d6cd47b79f585d2981821b97: No bootstrap required, opened a new log
I20240502 01:58:00.098837 7484 ts_tablet_manager.cc:1367] T 42d98987ce3e4e82ad0e55e00cd49c24 P 8b6a9ac9d6cd47b79f585d2981821b97: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240502 01:58:00.099108 7483 tablet_bootstrap.cc:492] T 42d98987ce3e4e82ad0e55e00cd49c24 P 5279c463ab5f4b6aa1d50c3fd0536174: No bootstrap required, opened a new log
I20240502 01:58:00.099175 7483 ts_tablet_manager.cc:1367] T 42d98987ce3e4e82ad0e55e00cd49c24 P 5279c463ab5f4b6aa1d50c3fd0536174: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240502 01:58:00.099196 7484 raft_consensus.cc:348] T 42d98987ce3e4e82ad0e55e00cd49c24 P 8b6a9ac9d6cd47b79f585d2981821b97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } }
I20240502 01:58:00.099273 7484 raft_consensus.cc:374] T 42d98987ce3e4e82ad0e55e00cd49c24 P 8b6a9ac9d6cd47b79f585d2981821b97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:00.099301 7484 raft_consensus.cc:729] T 42d98987ce3e4e82ad0e55e00cd49c24 P 8b6a9ac9d6cd47b79f585d2981821b97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b6a9ac9d6cd47b79f585d2981821b97, State: Initialized, Role: FOLLOWER
I20240502 01:58:00.099431 7484 consensus_queue.cc:260] T 42d98987ce3e4e82ad0e55e00cd49c24 P 8b6a9ac9d6cd47b79f585d2981821b97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } }
I20240502 01:58:00.099536 7483 raft_consensus.cc:348] T 42d98987ce3e4e82ad0e55e00cd49c24 P 5279c463ab5f4b6aa1d50c3fd0536174 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } }
I20240502 01:58:00.099575 7485 tablet_bootstrap.cc:654] T 7930116ca8d646faa009be917af87de8 P 97c52edf29a843bba2cd1653859f994e: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:00.099628 7483 raft_consensus.cc:374] T 42d98987ce3e4e82ad0e55e00cd49c24 P 5279c463ab5f4b6aa1d50c3fd0536174 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:00.099660 7483 raft_consensus.cc:729] T 42d98987ce3e4e82ad0e55e00cd49c24 P 5279c463ab5f4b6aa1d50c3fd0536174 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5279c463ab5f4b6aa1d50c3fd0536174, State: Initialized, Role: FOLLOWER
I20240502 01:58:00.099704 7484 ts_tablet_manager.cc:1398] T 42d98987ce3e4e82ad0e55e00cd49c24 P 8b6a9ac9d6cd47b79f585d2981821b97: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:00.099839 7484 tablet_bootstrap.cc:492] T 7930116ca8d646faa009be917af87de8 P 8b6a9ac9d6cd47b79f585d2981821b97: Bootstrap starting.
I20240502 01:58:00.099788 7483 consensus_queue.cc:260] T 42d98987ce3e4e82ad0e55e00cd49c24 P 5279c463ab5f4b6aa1d50c3fd0536174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } }
I20240502 01:58:00.100049 7483 ts_tablet_manager.cc:1398] T 42d98987ce3e4e82ad0e55e00cd49c24 P 5279c463ab5f4b6aa1d50c3fd0536174: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:00.100212 7483 tablet_bootstrap.cc:492] T ef99459b1bb145689ba08b01e28e4093 P 5279c463ab5f4b6aa1d50c3fd0536174: Bootstrap starting.
I20240502 01:58:00.101130 7485 tablet_bootstrap.cc:492] T 7930116ca8d646faa009be917af87de8 P 97c52edf29a843bba2cd1653859f994e: No bootstrap required, opened a new log
I20240502 01:58:00.101219 7485 ts_tablet_manager.cc:1367] T 7930116ca8d646faa009be917af87de8 P 97c52edf29a843bba2cd1653859f994e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:58:00.101457 7483 tablet_bootstrap.cc:654] T ef99459b1bb145689ba08b01e28e4093 P 5279c463ab5f4b6aa1d50c3fd0536174: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:00.101593 7485 raft_consensus.cc:348] T 7930116ca8d646faa009be917af87de8 P 97c52edf29a843bba2cd1653859f994e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } }
I20240502 01:58:00.101686 7485 raft_consensus.cc:374] T 7930116ca8d646faa009be917af87de8 P 97c52edf29a843bba2cd1653859f994e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:00.101716 7485 raft_consensus.cc:729] T 7930116ca8d646faa009be917af87de8 P 97c52edf29a843bba2cd1653859f994e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97c52edf29a843bba2cd1653859f994e, State: Initialized, Role: FOLLOWER
I20240502 01:58:00.101809 7485 consensus_queue.cc:260] T 7930116ca8d646faa009be917af87de8 P 97c52edf29a843bba2cd1653859f994e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } }
I20240502 01:58:00.102059 7485 ts_tablet_manager.cc:1398] T 7930116ca8d646faa009be917af87de8 P 97c52edf29a843bba2cd1653859f994e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:00.102152 7484 tablet_bootstrap.cc:654] T 7930116ca8d646faa009be917af87de8 P 8b6a9ac9d6cd47b79f585d2981821b97: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:00.102217 7485 tablet_bootstrap.cc:492] T e5668a318ea44db8a7c4db7fc7c95f7a P 97c52edf29a843bba2cd1653859f994e: Bootstrap starting.
I20240502 01:58:00.103010 7484 tablet_bootstrap.cc:492] T 7930116ca8d646faa009be917af87de8 P 8b6a9ac9d6cd47b79f585d2981821b97: No bootstrap required, opened a new log
I20240502 01:58:00.103057 7485 tablet_bootstrap.cc:654] T e5668a318ea44db8a7c4db7fc7c95f7a P 97c52edf29a843bba2cd1653859f994e: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:00.103086 7484 ts_tablet_manager.cc:1367] T 7930116ca8d646faa009be917af87de8 P 8b6a9ac9d6cd47b79f585d2981821b97: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:58:00.103420 7484 raft_consensus.cc:348] T 7930116ca8d646faa009be917af87de8 P 8b6a9ac9d6cd47b79f585d2981821b97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } }
I20240502 01:58:00.103514 7484 raft_consensus.cc:374] T 7930116ca8d646faa009be917af87de8 P 8b6a9ac9d6cd47b79f585d2981821b97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:00.103544 7484 raft_consensus.cc:729] T 7930116ca8d646faa009be917af87de8 P 8b6a9ac9d6cd47b79f585d2981821b97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b6a9ac9d6cd47b79f585d2981821b97, State: Initialized, Role: FOLLOWER
I20240502 01:58:00.103686 7484 consensus_queue.cc:260] T 7930116ca8d646faa009be917af87de8 P 8b6a9ac9d6cd47b79f585d2981821b97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } }
I20240502 01:58:00.103952 7485 tablet_bootstrap.cc:492] T e5668a318ea44db8a7c4db7fc7c95f7a P 97c52edf29a843bba2cd1653859f994e: No bootstrap required, opened a new log
I20240502 01:58:00.104027 7485 ts_tablet_manager.cc:1367] T e5668a318ea44db8a7c4db7fc7c95f7a P 97c52edf29a843bba2cd1653859f994e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:00.104333 7483 tablet_bootstrap.cc:492] T ef99459b1bb145689ba08b01e28e4093 P 5279c463ab5f4b6aa1d50c3fd0536174: No bootstrap required, opened a new log
I20240502 01:58:00.104410 7483 ts_tablet_manager.cc:1367] T ef99459b1bb145689ba08b01e28e4093 P 5279c463ab5f4b6aa1d50c3fd0536174: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:58:00.104457 7485 raft_consensus.cc:348] T e5668a318ea44db8a7c4db7fc7c95f7a P 97c52edf29a843bba2cd1653859f994e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } }
I20240502 01:58:00.104559 7485 raft_consensus.cc:374] T e5668a318ea44db8a7c4db7fc7c95f7a P 97c52edf29a843bba2cd1653859f994e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:00.104593 7485 raft_consensus.cc:729] T e5668a318ea44db8a7c4db7fc7c95f7a P 97c52edf29a843bba2cd1653859f994e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97c52edf29a843bba2cd1653859f994e, State: Initialized, Role: FOLLOWER
I20240502 01:58:00.104719 7485 consensus_queue.cc:260] T e5668a318ea44db8a7c4db7fc7c95f7a P 97c52edf29a843bba2cd1653859f994e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } }
I20240502 01:58:00.104782 7483 raft_consensus.cc:348] T ef99459b1bb145689ba08b01e28e4093 P 5279c463ab5f4b6aa1d50c3fd0536174 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } }
I20240502 01:58:00.104874 7483 raft_consensus.cc:374] T ef99459b1bb145689ba08b01e28e4093 P 5279c463ab5f4b6aa1d50c3fd0536174 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:00.104904 7483 raft_consensus.cc:729] T ef99459b1bb145689ba08b01e28e4093 P 5279c463ab5f4b6aa1d50c3fd0536174 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5279c463ab5f4b6aa1d50c3fd0536174, State: Initialized, Role: FOLLOWER
I20240502 01:58:00.105016 7483 consensus_queue.cc:260] T ef99459b1bb145689ba08b01e28e4093 P 5279c463ab5f4b6aa1d50c3fd0536174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } }
I20240502 01:58:00.105062 7485 ts_tablet_manager.cc:1398] T e5668a318ea44db8a7c4db7fc7c95f7a P 97c52edf29a843bba2cd1653859f994e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:00.105233 7485 tablet_bootstrap.cc:492] T 01d518fafee34875a7df792d8297834b P 97c52edf29a843bba2cd1653859f994e: Bootstrap starting.
I20240502 01:58:00.105233 7484 ts_tablet_manager.cc:1398] T 7930116ca8d646faa009be917af87de8 P 8b6a9ac9d6cd47b79f585d2981821b97: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:00.105312 7483 ts_tablet_manager.cc:1398] T ef99459b1bb145689ba08b01e28e4093 P 5279c463ab5f4b6aa1d50c3fd0536174: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:00.106241 7485 tablet_bootstrap.cc:654] T 01d518fafee34875a7df792d8297834b P 97c52edf29a843bba2cd1653859f994e: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:00.106904 7494 raft_consensus.cc:482] T ef99459b1bb145689ba08b01e28e4093 P 8b6a9ac9d6cd47b79f585d2981821b97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:00.106982 7494 raft_consensus.cc:504] T ef99459b1bb145689ba08b01e28e4093 P 8b6a9ac9d6cd47b79f585d2981821b97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } }
I20240502 01:58:00.107178 7485 tablet_bootstrap.cc:492] T 01d518fafee34875a7df792d8297834b P 97c52edf29a843bba2cd1653859f994e: No bootstrap required, opened a new log
I20240502 01:58:00.107262 7485 ts_tablet_manager.cc:1367] T 01d518fafee34875a7df792d8297834b P 97c52edf29a843bba2cd1653859f994e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:00.107309 7494 leader_election.cc:288] T ef99459b1bb145689ba08b01e28e4093 P 8b6a9ac9d6cd47b79f585d2981821b97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5279c463ab5f4b6aa1d50c3fd0536174 (127.6.117.193:44887), 97c52edf29a843bba2cd1653859f994e (127.6.117.194:44123)
I20240502 01:58:00.107614 7485 raft_consensus.cc:348] T 01d518fafee34875a7df792d8297834b P 97c52edf29a843bba2cd1653859f994e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } } peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } }
I20240502 01:58:00.107605 7361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef99459b1bb145689ba08b01e28e4093" candidate_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97c52edf29a843bba2cd1653859f994e" is_pre_election: true
I20240502 01:58:00.107606 7286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef99459b1bb145689ba08b01e28e4093" candidate_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" is_pre_election: true
I20240502 01:58:00.107725 7485 raft_consensus.cc:374] T 01d518fafee34875a7df792d8297834b P 97c52edf29a843bba2cd1653859f994e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:00.107756 7485 raft_consensus.cc:729] T 01d518fafee34875a7df792d8297834b P 97c52edf29a843bba2cd1653859f994e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97c52edf29a843bba2cd1653859f994e, State: Initialized, Role: FOLLOWER
I20240502 01:58:00.107764 7361 raft_consensus.cc:2454] T ef99459b1bb145689ba08b01e28e4093 P 97c52edf29a843bba2cd1653859f994e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b6a9ac9d6cd47b79f585d2981821b97 in term 0.
I20240502 01:58:00.107779 7286 raft_consensus.cc:2454] T ef99459b1bb145689ba08b01e28e4093 P 5279c463ab5f4b6aa1d50c3fd0536174 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b6a9ac9d6cd47b79f585d2981821b97 in term 0.
I20240502 01:58:00.107863 7485 consensus_queue.cc:260] T 01d518fafee34875a7df792d8297834b P 97c52edf29a843bba2cd1653859f994e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } } peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } }
I20240502 01:58:00.108031 7400 leader_election.cc:302] T ef99459b1bb145689ba08b01e28e4093 P 8b6a9ac9d6cd47b79f585d2981821b97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b6a9ac9d6cd47b79f585d2981821b97, 97c52edf29a843bba2cd1653859f994e; no voters:
I20240502 01:58:00.108165 7494 raft_consensus.cc:2781] T ef99459b1bb145689ba08b01e28e4093 P 8b6a9ac9d6cd47b79f585d2981821b97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:00.108213 7494 raft_consensus.cc:482] T ef99459b1bb145689ba08b01e28e4093 P 8b6a9ac9d6cd47b79f585d2981821b97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:00.108250 7494 raft_consensus.cc:3037] T ef99459b1bb145689ba08b01e28e4093 P 8b6a9ac9d6cd47b79f585d2981821b97 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:00.108361 7485 ts_tablet_manager.cc:1398] T 01d518fafee34875a7df792d8297834b P 97c52edf29a843bba2cd1653859f994e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:00.118219 7494 raft_consensus.cc:504] T ef99459b1bb145689ba08b01e28e4093 P 8b6a9ac9d6cd47b79f585d2981821b97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } }
I20240502 01:58:00.118616 7494 leader_election.cc:288] T ef99459b1bb145689ba08b01e28e4093 P 8b6a9ac9d6cd47b79f585d2981821b97 [CANDIDATE]: Term 1 election: Requested vote from peers 5279c463ab5f4b6aa1d50c3fd0536174 (127.6.117.193:44887), 97c52edf29a843bba2cd1653859f994e (127.6.117.194:44123)
I20240502 01:58:00.118973 7286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef99459b1bb145689ba08b01e28e4093" candidate_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5279c463ab5f4b6aa1d50c3fd0536174"
I20240502 01:58:00.118974 7361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef99459b1bb145689ba08b01e28e4093" candidate_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97c52edf29a843bba2cd1653859f994e"
I20240502 01:58:00.119092 7286 raft_consensus.cc:3037] T ef99459b1bb145689ba08b01e28e4093 P 5279c463ab5f4b6aa1d50c3fd0536174 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:00.119143 7361 raft_consensus.cc:3037] T ef99459b1bb145689ba08b01e28e4093 P 97c52edf29a843bba2cd1653859f994e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:00.125213 7361 raft_consensus.cc:2454] T ef99459b1bb145689ba08b01e28e4093 P 97c52edf29a843bba2cd1653859f994e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b6a9ac9d6cd47b79f585d2981821b97 in term 1.
I20240502 01:58:00.125213 7286 raft_consensus.cc:2454] T ef99459b1bb145689ba08b01e28e4093 P 5279c463ab5f4b6aa1d50c3fd0536174 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b6a9ac9d6cd47b79f585d2981821b97 in term 1.
I20240502 01:58:00.125533 7400 leader_election.cc:302] T ef99459b1bb145689ba08b01e28e4093 P 8b6a9ac9d6cd47b79f585d2981821b97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5279c463ab5f4b6aa1d50c3fd0536174, 8b6a9ac9d6cd47b79f585d2981821b97; no voters:
I20240502 01:58:00.125689 7494 raft_consensus.cc:2781] T ef99459b1bb145689ba08b01e28e4093 P 8b6a9ac9d6cd47b79f585d2981821b97 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:00.125777 7494 raft_consensus.cc:686] T ef99459b1bb145689ba08b01e28e4093 P 8b6a9ac9d6cd47b79f585d2981821b97 [term 1 LEADER]: Becoming Leader. State: Replica: 8b6a9ac9d6cd47b79f585d2981821b97, State: Running, Role: LEADER
I20240502 01:58:00.125912 7494 consensus_queue.cc:237] T ef99459b1bb145689ba08b01e28e4093 P 8b6a9ac9d6cd47b79f585d2981821b97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } }
I20240502 01:58:00.127478 7181 catalog_manager.cc:5484] T ef99459b1bb145689ba08b01e28e4093 P 8b6a9ac9d6cd47b79f585d2981821b97 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b6a9ac9d6cd47b79f585d2981821b97 (127.6.117.195). New cstate: current_term: 1 leader_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } health_report { overall_health: HEALTHY } } }
I20240502 01:58:00.134758 7498 raft_consensus.cc:482] T f3e2a90036a74f5a81d94a49595dc62a P 97c52edf29a843bba2cd1653859f994e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:00.134845 7498 raft_consensus.cc:504] T f3e2a90036a74f5a81d94a49595dc62a P 97c52edf29a843bba2cd1653859f994e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } }
I20240502 01:58:00.135267 7498 leader_election.cc:288] T f3e2a90036a74f5a81d94a49595dc62a P 97c52edf29a843bba2cd1653859f994e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5279c463ab5f4b6aa1d50c3fd0536174 (127.6.117.193:44887), 8b6a9ac9d6cd47b79f585d2981821b97 (127.6.117.195:36729)
I20240502 01:58:00.136871 7498 raft_consensus.cc:482] T 42d98987ce3e4e82ad0e55e00cd49c24 P 97c52edf29a843bba2cd1653859f994e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:00.136958 7498 raft_consensus.cc:504] T 42d98987ce3e4e82ad0e55e00cd49c24 P 97c52edf29a843bba2cd1653859f994e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } }
I20240502 01:58:00.137287 7498 leader_election.cc:288] T 42d98987ce3e4e82ad0e55e00cd49c24 P 97c52edf29a843bba2cd1653859f994e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5279c463ab5f4b6aa1d50c3fd0536174 (127.6.117.193:44887), 8b6a9ac9d6cd47b79f585d2981821b97 (127.6.117.195:36729)
I20240502 01:58:00.137822 7489 raft_consensus.cc:482] T 42d98987ce3e4e82ad0e55e00cd49c24 P 5279c463ab5f4b6aa1d50c3fd0536174 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:00.137897 7489 raft_consensus.cc:504] T 42d98987ce3e4e82ad0e55e00cd49c24 P 5279c463ab5f4b6aa1d50c3fd0536174 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } }
I20240502 01:58:00.138305 7489 leader_election.cc:288] T 42d98987ce3e4e82ad0e55e00cd49c24 P 5279c463ab5f4b6aa1d50c3fd0536174 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97c52edf29a843bba2cd1653859f994e (127.6.117.194:44123), 8b6a9ac9d6cd47b79f585d2981821b97 (127.6.117.195:36729)
I20240502 01:58:00.139245 7437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3e2a90036a74f5a81d94a49595dc62a" candidate_uuid: "97c52edf29a843bba2cd1653859f994e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" is_pre_election: true
I20240502 01:58:00.139377 7437 raft_consensus.cc:2454] T f3e2a90036a74f5a81d94a49595dc62a P 8b6a9ac9d6cd47b79f585d2981821b97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97c52edf29a843bba2cd1653859f994e in term 0.
I20240502 01:58:00.139433 7286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3e2a90036a74f5a81d94a49595dc62a" candidate_uuid: "97c52edf29a843bba2cd1653859f994e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" is_pre_election: true
I20240502 01:58:00.139551 7286 raft_consensus.cc:2454] T f3e2a90036a74f5a81d94a49595dc62a P 5279c463ab5f4b6aa1d50c3fd0536174 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97c52edf29a843bba2cd1653859f994e in term 0.
I20240502 01:58:00.139588 7285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42d98987ce3e4e82ad0e55e00cd49c24" candidate_uuid: "97c52edf29a843bba2cd1653859f994e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" is_pre_election: true
I20240502 01:58:00.139699 7285 raft_consensus.cc:2454] T 42d98987ce3e4e82ad0e55e00cd49c24 P 5279c463ab5f4b6aa1d50c3fd0536174 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97c52edf29a843bba2cd1653859f994e in term 0.
I20240502 01:58:00.139748 7436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42d98987ce3e4e82ad0e55e00cd49c24" candidate_uuid: "97c52edf29a843bba2cd1653859f994e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" is_pre_election: true
I20240502 01:58:00.139818 7325 leader_election.cc:302] T f3e2a90036a74f5a81d94a49595dc62a P 97c52edf29a843bba2cd1653859f994e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5279c463ab5f4b6aa1d50c3fd0536174, 97c52edf29a843bba2cd1653859f994e; no voters:
I20240502 01:58:00.139963 7498 raft_consensus.cc:2781] T f3e2a90036a74f5a81d94a49595dc62a P 97c52edf29a843bba2cd1653859f994e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:00.139993 7325 leader_election.cc:302] T 42d98987ce3e4e82ad0e55e00cd49c24 P 97c52edf29a843bba2cd1653859f994e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5279c463ab5f4b6aa1d50c3fd0536174, 97c52edf29a843bba2cd1653859f994e; no voters:
I20240502 01:58:00.140024 7498 raft_consensus.cc:482] T f3e2a90036a74f5a81d94a49595dc62a P 97c52edf29a843bba2cd1653859f994e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:00.140061 7498 raft_consensus.cc:3037] T f3e2a90036a74f5a81d94a49595dc62a P 97c52edf29a843bba2cd1653859f994e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:00.140129 7493 raft_consensus.cc:2781] T 42d98987ce3e4e82ad0e55e00cd49c24 P 97c52edf29a843bba2cd1653859f994e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:00.140165 7493 raft_consensus.cc:482] T 42d98987ce3e4e82ad0e55e00cd49c24 P 97c52edf29a843bba2cd1653859f994e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:00.140200 7493 raft_consensus.cc:3037] T 42d98987ce3e4e82ad0e55e00cd49c24 P 97c52edf29a843bba2cd1653859f994e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:00.141064 7493 raft_consensus.cc:504] T 42d98987ce3e4e82ad0e55e00cd49c24 P 97c52edf29a843bba2cd1653859f994e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } }
I20240502 01:58:00.139824 7436 raft_consensus.cc:2454] T 42d98987ce3e4e82ad0e55e00cd49c24 P 8b6a9ac9d6cd47b79f585d2981821b97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97c52edf29a843bba2cd1653859f994e in term 0.
I20240502 01:58:00.141383 7493 leader_election.cc:288] T 42d98987ce3e4e82ad0e55e00cd49c24 P 97c52edf29a843bba2cd1653859f994e [CANDIDATE]: Term 1 election: Requested vote from peers 5279c463ab5f4b6aa1d50c3fd0536174 (127.6.117.193:44887), 8b6a9ac9d6cd47b79f585d2981821b97 (127.6.117.195:36729)
I20240502 01:58:00.141744 7436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42d98987ce3e4e82ad0e55e00cd49c24" candidate_uuid: "97c52edf29a843bba2cd1653859f994e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b6a9ac9d6cd47b79f585d2981821b97"
I20240502 01:58:00.141796 7285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42d98987ce3e4e82ad0e55e00cd49c24" candidate_uuid: "97c52edf29a843bba2cd1653859f994e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5279c463ab5f4b6aa1d50c3fd0536174"
I20240502 01:58:00.141858 7436 raft_consensus.cc:3037] T 42d98987ce3e4e82ad0e55e00cd49c24 P 8b6a9ac9d6cd47b79f585d2981821b97 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:00.141891 7285 raft_consensus.cc:3037] T 42d98987ce3e4e82ad0e55e00cd49c24 P 5279c463ab5f4b6aa1d50c3fd0536174 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:00.142827 7436 raft_consensus.cc:2454] T 42d98987ce3e4e82ad0e55e00cd49c24 P 8b6a9ac9d6cd47b79f585d2981821b97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97c52edf29a843bba2cd1653859f994e in term 1.
I20240502 01:58:00.143096 7328 leader_election.cc:302] T 42d98987ce3e4e82ad0e55e00cd49c24 P 97c52edf29a843bba2cd1653859f994e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b6a9ac9d6cd47b79f585d2981821b97, 97c52edf29a843bba2cd1653859f994e; no voters:
I20240502 01:58:00.143539 7493 raft_consensus.cc:2781] T 42d98987ce3e4e82ad0e55e00cd49c24 P 97c52edf29a843bba2cd1653859f994e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:00.142807 7285 raft_consensus.cc:2454] T 42d98987ce3e4e82ad0e55e00cd49c24 P 5279c463ab5f4b6aa1d50c3fd0536174 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97c52edf29a843bba2cd1653859f994e in term 1.
I20240502 01:58:00.143718 7361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42d98987ce3e4e82ad0e55e00cd49c24" candidate_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97c52edf29a843bba2cd1653859f994e" is_pre_election: true
I20240502 01:58:00.143893 7436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42d98987ce3e4e82ad0e55e00cd49c24" candidate_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" is_pre_election: true
I20240502 01:58:00.143724 7493 raft_consensus.cc:686] T 42d98987ce3e4e82ad0e55e00cd49c24 P 97c52edf29a843bba2cd1653859f994e [term 1 LEADER]: Becoming Leader. State: Replica: 97c52edf29a843bba2cd1653859f994e, State: Running, Role: LEADER
I20240502 01:58:00.144053 7436 raft_consensus.cc:2379] T 42d98987ce3e4e82ad0e55e00cd49c24 P 8b6a9ac9d6cd47b79f585d2981821b97 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5279c463ab5f4b6aa1d50c3fd0536174 in current term 1: Already voted for candidate 97c52edf29a843bba2cd1653859f994e in this term.
I20240502 01:58:00.141124 7498 raft_consensus.cc:504] T f3e2a90036a74f5a81d94a49595dc62a P 97c52edf29a843bba2cd1653859f994e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } }
I20240502 01:58:00.144086 7493 consensus_queue.cc:237] T 42d98987ce3e4e82ad0e55e00cd49c24 P 97c52edf29a843bba2cd1653859f994e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } }
I20240502 01:58:00.144440 7498 leader_election.cc:288] T f3e2a90036a74f5a81d94a49595dc62a P 97c52edf29a843bba2cd1653859f994e [CANDIDATE]: Term 1 election: Requested vote from peers 5279c463ab5f4b6aa1d50c3fd0536174 (127.6.117.193:44887), 8b6a9ac9d6cd47b79f585d2981821b97 (127.6.117.195:36729)
I20240502 01:58:00.144651 7285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3e2a90036a74f5a81d94a49595dc62a" candidate_uuid: "97c52edf29a843bba2cd1653859f994e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5279c463ab5f4b6aa1d50c3fd0536174"
I20240502 01:58:00.144753 7436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3e2a90036a74f5a81d94a49595dc62a" candidate_uuid: "97c52edf29a843bba2cd1653859f994e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b6a9ac9d6cd47b79f585d2981821b97"
I20240502 01:58:00.144753 7285 raft_consensus.cc:3037] T f3e2a90036a74f5a81d94a49595dc62a P 5279c463ab5f4b6aa1d50c3fd0536174 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:00.144877 7436 raft_consensus.cc:3037] T f3e2a90036a74f5a81d94a49595dc62a P 8b6a9ac9d6cd47b79f585d2981821b97 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:00.145381 7249 leader_election.cc:302] T 42d98987ce3e4e82ad0e55e00cd49c24 P 5279c463ab5f4b6aa1d50c3fd0536174 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5279c463ab5f4b6aa1d50c3fd0536174; no voters: 8b6a9ac9d6cd47b79f585d2981821b97, 97c52edf29a843bba2cd1653859f994e
I20240502 01:58:00.145581 7489 raft_consensus.cc:2726] T 42d98987ce3e4e82ad0e55e00cd49c24 P 5279c463ab5f4b6aa1d50c3fd0536174 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:58:00.145633 7436 raft_consensus.cc:2454] T f3e2a90036a74f5a81d94a49595dc62a P 8b6a9ac9d6cd47b79f585d2981821b97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97c52edf29a843bba2cd1653859f994e in term 1.
I20240502 01:58:00.145642 7285 raft_consensus.cc:2454] T f3e2a90036a74f5a81d94a49595dc62a P 5279c463ab5f4b6aa1d50c3fd0536174 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97c52edf29a843bba2cd1653859f994e in term 1.
I20240502 01:58:00.145948 7325 leader_election.cc:302] T f3e2a90036a74f5a81d94a49595dc62a P 97c52edf29a843bba2cd1653859f994e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5279c463ab5f4b6aa1d50c3fd0536174, 97c52edf29a843bba2cd1653859f994e; no voters:
I20240502 01:58:00.146142 7493 raft_consensus.cc:2781] T f3e2a90036a74f5a81d94a49595dc62a P 97c52edf29a843bba2cd1653859f994e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:00.146212 7493 raft_consensus.cc:686] T f3e2a90036a74f5a81d94a49595dc62a P 97c52edf29a843bba2cd1653859f994e [term 1 LEADER]: Becoming Leader. State: Replica: 97c52edf29a843bba2cd1653859f994e, State: Running, Role: LEADER
I20240502 01:58:00.146327 7493 consensus_queue.cc:237] T f3e2a90036a74f5a81d94a49595dc62a P 97c52edf29a843bba2cd1653859f994e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } }
I20240502 01:58:00.146278 7181 catalog_manager.cc:5484] T 42d98987ce3e4e82ad0e55e00cd49c24 P 97c52edf29a843bba2cd1653859f994e reported cstate change: term changed from 0 to 1, leader changed from <none> to 97c52edf29a843bba2cd1653859f994e (127.6.117.194). New cstate: current_term: 1 leader_uuid: "97c52edf29a843bba2cd1653859f994e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:00.149179 7181 catalog_manager.cc:5484] T f3e2a90036a74f5a81d94a49595dc62a P 97c52edf29a843bba2cd1653859f994e reported cstate change: term changed from 0 to 1, leader changed from <none> to 97c52edf29a843bba2cd1653859f994e (127.6.117.194). New cstate: current_term: 1 leader_uuid: "97c52edf29a843bba2cd1653859f994e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:00.164297 7494 raft_consensus.cc:482] T c1ce57f32584460f9bd94d85d19c8c4d P 8b6a9ac9d6cd47b79f585d2981821b97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:00.164458 7494 raft_consensus.cc:504] T c1ce57f32584460f9bd94d85d19c8c4d P 8b6a9ac9d6cd47b79f585d2981821b97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } }
I20240502 01:58:00.164848 7494 leader_election.cc:288] T c1ce57f32584460f9bd94d85d19c8c4d P 8b6a9ac9d6cd47b79f585d2981821b97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5279c463ab5f4b6aa1d50c3fd0536174 (127.6.117.193:44887), 97c52edf29a843bba2cd1653859f994e (127.6.117.194:44123)
I20240502 01:58:00.165287 7285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1ce57f32584460f9bd94d85d19c8c4d" candidate_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" is_pre_election: true
I20240502 01:58:00.165289 7361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1ce57f32584460f9bd94d85d19c8c4d" candidate_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97c52edf29a843bba2cd1653859f994e" is_pre_election: true
I20240502 01:58:00.165427 7285 raft_consensus.cc:2454] T c1ce57f32584460f9bd94d85d19c8c4d P 5279c463ab5f4b6aa1d50c3fd0536174 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b6a9ac9d6cd47b79f585d2981821b97 in term 0.
I20240502 01:58:00.165493 7361 raft_consensus.cc:2454] T c1ce57f32584460f9bd94d85d19c8c4d P 97c52edf29a843bba2cd1653859f994e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b6a9ac9d6cd47b79f585d2981821b97 in term 0.
I20240502 01:58:00.165694 7400 leader_election.cc:302] T c1ce57f32584460f9bd94d85d19c8c4d P 8b6a9ac9d6cd47b79f585d2981821b97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5279c463ab5f4b6aa1d50c3fd0536174, 8b6a9ac9d6cd47b79f585d2981821b97; no voters:
I20240502 01:58:00.165845 7494 raft_consensus.cc:2781] T c1ce57f32584460f9bd94d85d19c8c4d P 8b6a9ac9d6cd47b79f585d2981821b97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:00.165905 7494 raft_consensus.cc:482] T c1ce57f32584460f9bd94d85d19c8c4d P 8b6a9ac9d6cd47b79f585d2981821b97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:00.165936 7494 raft_consensus.cc:3037] T c1ce57f32584460f9bd94d85d19c8c4d P 8b6a9ac9d6cd47b79f585d2981821b97 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:00.166692 7494 raft_consensus.cc:504] T c1ce57f32584460f9bd94d85d19c8c4d P 8b6a9ac9d6cd47b79f585d2981821b97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } }
I20240502 01:58:00.166971 7494 leader_election.cc:288] T c1ce57f32584460f9bd94d85d19c8c4d P 8b6a9ac9d6cd47b79f585d2981821b97 [CANDIDATE]: Term 1 election: Requested vote from peers 5279c463ab5f4b6aa1d50c3fd0536174 (127.6.117.193:44887), 97c52edf29a843bba2cd1653859f994e (127.6.117.194:44123)
I20240502 01:58:00.167246 7361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1ce57f32584460f9bd94d85d19c8c4d" candidate_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97c52edf29a843bba2cd1653859f994e"
I20240502 01:58:00.167269 7285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1ce57f32584460f9bd94d85d19c8c4d" candidate_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5279c463ab5f4b6aa1d50c3fd0536174"
I20240502 01:58:00.167333 7361 raft_consensus.cc:3037] T c1ce57f32584460f9bd94d85d19c8c4d P 97c52edf29a843bba2cd1653859f994e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:00.167362 7285 raft_consensus.cc:3037] T c1ce57f32584460f9bd94d85d19c8c4d P 5279c463ab5f4b6aa1d50c3fd0536174 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:00.167986 7285 raft_consensus.cc:2454] T c1ce57f32584460f9bd94d85d19c8c4d P 5279c463ab5f4b6aa1d50c3fd0536174 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b6a9ac9d6cd47b79f585d2981821b97 in term 1.
I20240502 01:58:00.168040 7361 raft_consensus.cc:2454] T c1ce57f32584460f9bd94d85d19c8c4d P 97c52edf29a843bba2cd1653859f994e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b6a9ac9d6cd47b79f585d2981821b97 in term 1.
I20240502 01:58:00.168213 7400 leader_election.cc:302] T c1ce57f32584460f9bd94d85d19c8c4d P 8b6a9ac9d6cd47b79f585d2981821b97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5279c463ab5f4b6aa1d50c3fd0536174, 8b6a9ac9d6cd47b79f585d2981821b97; no voters:
I20240502 01:58:00.168498 7494 raft_consensus.cc:2781] T c1ce57f32584460f9bd94d85d19c8c4d P 8b6a9ac9d6cd47b79f585d2981821b97 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:00.168577 7494 raft_consensus.cc:686] T c1ce57f32584460f9bd94d85d19c8c4d P 8b6a9ac9d6cd47b79f585d2981821b97 [term 1 LEADER]: Becoming Leader. State: Replica: 8b6a9ac9d6cd47b79f585d2981821b97, State: Running, Role: LEADER
I20240502 01:58:00.168707 7494 consensus_queue.cc:237] T c1ce57f32584460f9bd94d85d19c8c4d P 8b6a9ac9d6cd47b79f585d2981821b97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } }
I20240502 01:58:00.170102 7181 catalog_manager.cc:5484] T c1ce57f32584460f9bd94d85d19c8c4d P 8b6a9ac9d6cd47b79f585d2981821b97 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b6a9ac9d6cd47b79f585d2981821b97 (127.6.117.195). New cstate: current_term: 1 leader_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } health_report { overall_health: HEALTHY } } }
I20240502 01:58:00.208477 7494 raft_consensus.cc:482] T 7ba4aedf729d49bc977d5e52af888ebc P 8b6a9ac9d6cd47b79f585d2981821b97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:00.208626 7494 raft_consensus.cc:504] T 7ba4aedf729d49bc977d5e52af888ebc P 8b6a9ac9d6cd47b79f585d2981821b97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } }
I20240502 01:58:00.209020 7494 leader_election.cc:288] T 7ba4aedf729d49bc977d5e52af888ebc P 8b6a9ac9d6cd47b79f585d2981821b97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5279c463ab5f4b6aa1d50c3fd0536174 (127.6.117.193:44887), 97c52edf29a843bba2cd1653859f994e (127.6.117.194:44123)
I20240502 01:58:00.209435 7285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ba4aedf729d49bc977d5e52af888ebc" candidate_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" is_pre_election: true
I20240502 01:58:00.209475 7361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ba4aedf729d49bc977d5e52af888ebc" candidate_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97c52edf29a843bba2cd1653859f994e" is_pre_election: true
I20240502 01:58:00.209597 7361 raft_consensus.cc:2454] T 7ba4aedf729d49bc977d5e52af888ebc P 97c52edf29a843bba2cd1653859f994e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b6a9ac9d6cd47b79f585d2981821b97 in term 0.
I20240502 01:58:00.209553 7285 raft_consensus.cc:2454] T 7ba4aedf729d49bc977d5e52af888ebc P 5279c463ab5f4b6aa1d50c3fd0536174 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b6a9ac9d6cd47b79f585d2981821b97 in term 0.
I20240502 01:58:00.209882 7400 leader_election.cc:302] T 7ba4aedf729d49bc977d5e52af888ebc P 8b6a9ac9d6cd47b79f585d2981821b97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b6a9ac9d6cd47b79f585d2981821b97, 97c52edf29a843bba2cd1653859f994e; no voters:
I20240502 01:58:00.210036 7494 raft_consensus.cc:2781] T 7ba4aedf729d49bc977d5e52af888ebc P 8b6a9ac9d6cd47b79f585d2981821b97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:00.210093 7494 raft_consensus.cc:482] T 7ba4aedf729d49bc977d5e52af888ebc P 8b6a9ac9d6cd47b79f585d2981821b97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:00.210121 7494 raft_consensus.cc:3037] T 7ba4aedf729d49bc977d5e52af888ebc P 8b6a9ac9d6cd47b79f585d2981821b97 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:00.210897 7494 raft_consensus.cc:504] T 7ba4aedf729d49bc977d5e52af888ebc P 8b6a9ac9d6cd47b79f585d2981821b97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } }
I20240502 01:58:00.211176 7494 leader_election.cc:288] T 7ba4aedf729d49bc977d5e52af888ebc P 8b6a9ac9d6cd47b79f585d2981821b97 [CANDIDATE]: Term 1 election: Requested vote from peers 5279c463ab5f4b6aa1d50c3fd0536174 (127.6.117.193:44887), 97c52edf29a843bba2cd1653859f994e (127.6.117.194:44123)
I20240502 01:58:00.211445 7285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ba4aedf729d49bc977d5e52af888ebc" candidate_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5279c463ab5f4b6aa1d50c3fd0536174"
I20240502 01:58:00.211459 7361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ba4aedf729d49bc977d5e52af888ebc" candidate_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97c52edf29a843bba2cd1653859f994e"
I20240502 01:58:00.211544 7285 raft_consensus.cc:3037] T 7ba4aedf729d49bc977d5e52af888ebc P 5279c463ab5f4b6aa1d50c3fd0536174 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:00.211596 7361 raft_consensus.cc:3037] T 7ba4aedf729d49bc977d5e52af888ebc P 97c52edf29a843bba2cd1653859f994e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:00.212306 7285 raft_consensus.cc:2454] T 7ba4aedf729d49bc977d5e52af888ebc P 5279c463ab5f4b6aa1d50c3fd0536174 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b6a9ac9d6cd47b79f585d2981821b97 in term 1.
I20240502 01:58:00.212308 7361 raft_consensus.cc:2454] T 7ba4aedf729d49bc977d5e52af888ebc P 97c52edf29a843bba2cd1653859f994e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b6a9ac9d6cd47b79f585d2981821b97 in term 1.
I20240502 01:58:00.212574 7400 leader_election.cc:302] T 7ba4aedf729d49bc977d5e52af888ebc P 8b6a9ac9d6cd47b79f585d2981821b97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b6a9ac9d6cd47b79f585d2981821b97, 97c52edf29a843bba2cd1653859f994e; no voters:
I20240502 01:58:00.212738 7494 raft_consensus.cc:2781] T 7ba4aedf729d49bc977d5e52af888ebc P 8b6a9ac9d6cd47b79f585d2981821b97 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:00.212833 7494 raft_consensus.cc:686] T 7ba4aedf729d49bc977d5e52af888ebc P 8b6a9ac9d6cd47b79f585d2981821b97 [term 1 LEADER]: Becoming Leader. State: Replica: 8b6a9ac9d6cd47b79f585d2981821b97, State: Running, Role: LEADER
I20240502 01:58:00.212967 7494 consensus_queue.cc:237] T 7ba4aedf729d49bc977d5e52af888ebc P 8b6a9ac9d6cd47b79f585d2981821b97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } }
I20240502 01:58:00.214411 7181 catalog_manager.cc:5484] T 7ba4aedf729d49bc977d5e52af888ebc P 8b6a9ac9d6cd47b79f585d2981821b97 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b6a9ac9d6cd47b79f585d2981821b97 (127.6.117.195). New cstate: current_term: 1 leader_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } health_report { overall_health: HEALTHY } } }
I20240502 01:58:00.355521 7493 raft_consensus.cc:482] T 7930116ca8d646faa009be917af87de8 P 97c52edf29a843bba2cd1653859f994e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:00.355628 7494 raft_consensus.cc:482] T e5668a318ea44db8a7c4db7fc7c95f7a P 8b6a9ac9d6cd47b79f585d2981821b97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:00.355686 7493 raft_consensus.cc:504] T 7930116ca8d646faa009be917af87de8 P 97c52edf29a843bba2cd1653859f994e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } }
I20240502 01:58:00.355731 7494 raft_consensus.cc:504] T e5668a318ea44db8a7c4db7fc7c95f7a P 8b6a9ac9d6cd47b79f585d2981821b97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } }
I20240502 01:58:00.356115 7493 leader_election.cc:288] T 7930116ca8d646faa009be917af87de8 P 97c52edf29a843bba2cd1653859f994e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5279c463ab5f4b6aa1d50c3fd0536174 (127.6.117.193:44887), 8b6a9ac9d6cd47b79f585d2981821b97 (127.6.117.195:36729)
I20240502 01:58:00.356106 7494 leader_election.cc:288] T e5668a318ea44db8a7c4db7fc7c95f7a P 8b6a9ac9d6cd47b79f585d2981821b97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5279c463ab5f4b6aa1d50c3fd0536174 (127.6.117.193:44887), 97c52edf29a843bba2cd1653859f994e (127.6.117.194:44123)
I20240502 01:58:00.356516 7436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7930116ca8d646faa009be917af87de8" candidate_uuid: "97c52edf29a843bba2cd1653859f994e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" is_pre_election: true
I20240502 01:58:00.356585 7285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5668a318ea44db8a7c4db7fc7c95f7a" candidate_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" is_pre_election: true
I20240502 01:58:00.356516 7286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7930116ca8d646faa009be917af87de8" candidate_uuid: "97c52edf29a843bba2cd1653859f994e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" is_pre_election: true
I20240502 01:58:00.356666 7436 raft_consensus.cc:2454] T 7930116ca8d646faa009be917af87de8 P 8b6a9ac9d6cd47b79f585d2981821b97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97c52edf29a843bba2cd1653859f994e in term 0.
I20240502 01:58:00.356709 7285 raft_consensus.cc:2454] T e5668a318ea44db8a7c4db7fc7c95f7a P 5279c463ab5f4b6aa1d50c3fd0536174 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b6a9ac9d6cd47b79f585d2981821b97 in term 0.
I20240502 01:58:00.356726 7286 raft_consensus.cc:2454] T 7930116ca8d646faa009be917af87de8 P 5279c463ab5f4b6aa1d50c3fd0536174 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97c52edf29a843bba2cd1653859f994e in term 0.
I20240502 01:58:00.356516 7361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5668a318ea44db8a7c4db7fc7c95f7a" candidate_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97c52edf29a843bba2cd1653859f994e" is_pre_election: true
I20240502 01:58:00.356940 7361 raft_consensus.cc:2454] T e5668a318ea44db8a7c4db7fc7c95f7a P 97c52edf29a843bba2cd1653859f994e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b6a9ac9d6cd47b79f585d2981821b97 in term 0.
I20240502 01:58:00.356950 7325 leader_election.cc:302] T 7930116ca8d646faa009be917af87de8 P 97c52edf29a843bba2cd1653859f994e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5279c463ab5f4b6aa1d50c3fd0536174, 97c52edf29a843bba2cd1653859f994e; no voters:
I20240502 01:58:00.356997 7400 leader_election.cc:302] T e5668a318ea44db8a7c4db7fc7c95f7a P 8b6a9ac9d6cd47b79f585d2981821b97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5279c463ab5f4b6aa1d50c3fd0536174, 8b6a9ac9d6cd47b79f585d2981821b97; no voters:
I20240502 01:58:00.357124 7493 raft_consensus.cc:2781] T 7930116ca8d646faa009be917af87de8 P 97c52edf29a843bba2cd1653859f994e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:00.357170 7493 raft_consensus.cc:482] T 7930116ca8d646faa009be917af87de8 P 97c52edf29a843bba2cd1653859f994e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:00.357184 7494 raft_consensus.cc:2781] T e5668a318ea44db8a7c4db7fc7c95f7a P 8b6a9ac9d6cd47b79f585d2981821b97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:00.357206 7493 raft_consensus.cc:3037] T 7930116ca8d646faa009be917af87de8 P 97c52edf29a843bba2cd1653859f994e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:00.357221 7494 raft_consensus.cc:482] T e5668a318ea44db8a7c4db7fc7c95f7a P 8b6a9ac9d6cd47b79f585d2981821b97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:00.357261 7494 raft_consensus.cc:3037] T e5668a318ea44db8a7c4db7fc7c95f7a P 8b6a9ac9d6cd47b79f585d2981821b97 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:00.358177 7493 raft_consensus.cc:504] T 7930116ca8d646faa009be917af87de8 P 97c52edf29a843bba2cd1653859f994e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } }
I20240502 01:58:00.358177 7494 raft_consensus.cc:504] T e5668a318ea44db8a7c4db7fc7c95f7a P 8b6a9ac9d6cd47b79f585d2981821b97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } }
I20240502 01:58:00.358559 7493 leader_election.cc:288] T 7930116ca8d646faa009be917af87de8 P 97c52edf29a843bba2cd1653859f994e [CANDIDATE]: Term 1 election: Requested vote from peers 5279c463ab5f4b6aa1d50c3fd0536174 (127.6.117.193:44887), 8b6a9ac9d6cd47b79f585d2981821b97 (127.6.117.195:36729)
I20240502 01:58:00.358551 7494 leader_election.cc:288] T e5668a318ea44db8a7c4db7fc7c95f7a P 8b6a9ac9d6cd47b79f585d2981821b97 [CANDIDATE]: Term 1 election: Requested vote from peers 5279c463ab5f4b6aa1d50c3fd0536174 (127.6.117.193:44887), 97c52edf29a843bba2cd1653859f994e (127.6.117.194:44123)
I20240502 01:58:00.358796 7285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7930116ca8d646faa009be917af87de8" candidate_uuid: "97c52edf29a843bba2cd1653859f994e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5279c463ab5f4b6aa1d50c3fd0536174"
I20240502 01:58:00.358824 7286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5668a318ea44db8a7c4db7fc7c95f7a" candidate_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5279c463ab5f4b6aa1d50c3fd0536174"
I20240502 01:58:00.358821 7436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7930116ca8d646faa009be917af87de8" candidate_uuid: "97c52edf29a843bba2cd1653859f994e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b6a9ac9d6cd47b79f585d2981821b97"
I20240502 01:58:00.358857 7361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5668a318ea44db8a7c4db7fc7c95f7a" candidate_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97c52edf29a843bba2cd1653859f994e"
I20240502 01:58:00.358937 7285 raft_consensus.cc:3037] T 7930116ca8d646faa009be917af87de8 P 5279c463ab5f4b6aa1d50c3fd0536174 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:00.358937 7286 raft_consensus.cc:3037] T e5668a318ea44db8a7c4db7fc7c95f7a P 5279c463ab5f4b6aa1d50c3fd0536174 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:00.358978 7436 raft_consensus.cc:3037] T 7930116ca8d646faa009be917af87de8 P 8b6a9ac9d6cd47b79f585d2981821b97 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:00.358986 7361 raft_consensus.cc:3037] T e5668a318ea44db8a7c4db7fc7c95f7a P 97c52edf29a843bba2cd1653859f994e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:00.359766 7286 raft_consensus.cc:2454] T e5668a318ea44db8a7c4db7fc7c95f7a P 5279c463ab5f4b6aa1d50c3fd0536174 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b6a9ac9d6cd47b79f585d2981821b97 in term 1.
I20240502 01:58:00.359759 7361 raft_consensus.cc:2454] T e5668a318ea44db8a7c4db7fc7c95f7a P 97c52edf29a843bba2cd1653859f994e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b6a9ac9d6cd47b79f585d2981821b97 in term 1.
I20240502 01:58:00.359759 7436 raft_consensus.cc:2454] T 7930116ca8d646faa009be917af87de8 P 8b6a9ac9d6cd47b79f585d2981821b97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97c52edf29a843bba2cd1653859f994e in term 1.
I20240502 01:58:00.359822 7285 raft_consensus.cc:2454] T 7930116ca8d646faa009be917af87de8 P 5279c463ab5f4b6aa1d50c3fd0536174 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97c52edf29a843bba2cd1653859f994e in term 1.
I20240502 01:58:00.360040 7400 leader_election.cc:302] T e5668a318ea44db8a7c4db7fc7c95f7a P 8b6a9ac9d6cd47b79f585d2981821b97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5279c463ab5f4b6aa1d50c3fd0536174, 8b6a9ac9d6cd47b79f585d2981821b97; no voters:
I20240502 01:58:00.360121 7328 leader_election.cc:302] T 7930116ca8d646faa009be917af87de8 P 97c52edf29a843bba2cd1653859f994e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b6a9ac9d6cd47b79f585d2981821b97, 97c52edf29a843bba2cd1653859f994e; no voters:
I20240502 01:58:00.360183 7494 raft_consensus.cc:2781] T e5668a318ea44db8a7c4db7fc7c95f7a P 8b6a9ac9d6cd47b79f585d2981821b97 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:00.360283 7493 raft_consensus.cc:2781] T 7930116ca8d646faa009be917af87de8 P 97c52edf29a843bba2cd1653859f994e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:00.360302 7494 raft_consensus.cc:686] T e5668a318ea44db8a7c4db7fc7c95f7a P 8b6a9ac9d6cd47b79f585d2981821b97 [term 1 LEADER]: Becoming Leader. State: Replica: 8b6a9ac9d6cd47b79f585d2981821b97, State: Running, Role: LEADER
I20240502 01:58:00.360350 7493 raft_consensus.cc:686] T 7930116ca8d646faa009be917af87de8 P 97c52edf29a843bba2cd1653859f994e [term 1 LEADER]: Becoming Leader. State: Replica: 97c52edf29a843bba2cd1653859f994e, State: Running, Role: LEADER
I20240502 01:58:00.360453 7493 consensus_queue.cc:237] T 7930116ca8d646faa009be917af87de8 P 97c52edf29a843bba2cd1653859f994e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } }
I20240502 01:58:00.360463 7494 consensus_queue.cc:237] T e5668a318ea44db8a7c4db7fc7c95f7a P 8b6a9ac9d6cd47b79f585d2981821b97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } }
I20240502 01:58:00.362079 7181 catalog_manager.cc:5484] T 7930116ca8d646faa009be917af87de8 P 97c52edf29a843bba2cd1653859f994e reported cstate change: term changed from 0 to 1, leader changed from <none> to 97c52edf29a843bba2cd1653859f994e (127.6.117.194). New cstate: current_term: 1 leader_uuid: "97c52edf29a843bba2cd1653859f994e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:00.362339 7183 catalog_manager.cc:5484] T e5668a318ea44db8a7c4db7fc7c95f7a P 8b6a9ac9d6cd47b79f585d2981821b97 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b6a9ac9d6cd47b79f585d2981821b97 (127.6.117.195). New cstate: current_term: 1 leader_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } health_report { overall_health: HEALTHY } } }
I20240502 01:58:00.406713 6615 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:58:00.408618 7535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:00.409006 7533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:58:00.409371 6615 server_base.cc:1034] running on GCE node
W20240502 01:58:00.409672 7532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:58:00.409865 6615 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:58:00.409916 6615 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:58:00.409936 6615 hybrid_clock.cc:648] HybridClock initialized: now 1714615080409936 us; error 0 us; skew 500 ppm
I20240502 01:58:00.410928 6615 webserver.cc:460] Webserver started at http://127.6.117.196:33611/ using document root <none> and password file <none>
I20240502 01:58:00.411087 6615 fs_manager.cc:359] Metadata directory not provided
I20240502 01:58:00.411134 6615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:58:00.411211 6615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:58:00.411581 6615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714615076189642-6615-0/minicluster-data/ts-3-root/instance:
uuid: "a1c5c6906f434cea9d1966623e1ff641"
format_stamp: "Formatted at 2024-05-02 01:58:00 on dist-test-slave-89cm"
I20240502 01:58:00.413161 6615 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240502 01:58:00.414023 7540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:00.414253 6615 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:00.414317 6615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714615076189642-6615-0/minicluster-data/ts-3-root
uuid: "a1c5c6906f434cea9d1966623e1ff641"
format_stamp: "Formatted at 2024-05-02 01:58:00 on dist-test-slave-89cm"
I20240502 01:58:00.414382 6615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714615076189642-6615-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714615076189642-6615-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714615076189642-6615-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:58:00.426532 6615 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:58:00.426908 6615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:58:00.427206 6615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:58:00.427645 6615 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:58:00.427685 6615 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:00.427727 6615 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:58:00.427754 6615 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:00.432575 6615 rpc_server.cc:307] RPC server started. Bound to: 127.6.117.196:34745
I20240502 01:58:00.432622 7603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.117.196:34745 every 8 connection(s)
I20240502 01:58:00.436960 7604 heartbeater.cc:347] Connected to a master server at 127.6.117.254:35435
I20240502 01:58:00.437065 7604 heartbeater.cc:464] Registering TS with master...
I20240502 01:58:00.437099 7493 raft_consensus.cc:482] T 01d518fafee34875a7df792d8297834b P 97c52edf29a843bba2cd1653859f994e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:00.437206 7493 raft_consensus.cc:504] T 01d518fafee34875a7df792d8297834b P 97c52edf29a843bba2cd1653859f994e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } } peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } }
I20240502 01:58:00.437291 7604 heartbeater.cc:511] Master 127.6.117.254:35435 requested a full tablet report, sending...
I20240502 01:58:00.437551 7493 leader_election.cc:288] T 01d518fafee34875a7df792d8297834b P 97c52edf29a843bba2cd1653859f994e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b6a9ac9d6cd47b79f585d2981821b97 (127.6.117.195:36729), 5279c463ab5f4b6aa1d50c3fd0536174 (127.6.117.193:44887)
I20240502 01:58:00.437866 7285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01d518fafee34875a7df792d8297834b" candidate_uuid: "97c52edf29a843bba2cd1653859f994e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" is_pre_election: true
I20240502 01:58:00.437875 7436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01d518fafee34875a7df792d8297834b" candidate_uuid: "97c52edf29a843bba2cd1653859f994e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" is_pre_election: true
I20240502 01:58:00.437894 7183 ts_manager.cc:194] Registered new tserver with Master: a1c5c6906f434cea9d1966623e1ff641 (127.6.117.196:34745)
I20240502 01:58:00.438015 7285 raft_consensus.cc:2379] T 01d518fafee34875a7df792d8297834b P 5279c463ab5f4b6aa1d50c3fd0536174 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 97c52edf29a843bba2cd1653859f994e in current term 1: Already voted for candidate 8b6a9ac9d6cd47b79f585d2981821b97 in this term.
I20240502 01:58:00.438297 7325 leader_election.cc:302] T 01d518fafee34875a7df792d8297834b P 97c52edf29a843bba2cd1653859f994e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97c52edf29a843bba2cd1653859f994e; no voters: 5279c463ab5f4b6aa1d50c3fd0536174, 8b6a9ac9d6cd47b79f585d2981821b97
I20240502 01:58:00.438463 7493 raft_consensus.cc:3037] T 01d518fafee34875a7df792d8297834b P 97c52edf29a843bba2cd1653859f994e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:00.438661 7183 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56866
I20240502 01:58:00.439319 7493 raft_consensus.cc:2726] T 01d518fafee34875a7df792d8297834b P 97c52edf29a843bba2cd1653859f994e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:58:00.528146 7505 consensus_queue.cc:1035] T ef99459b1bb145689ba08b01e28e4093 P 8b6a9ac9d6cd47b79f585d2981821b97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:00.529031 7505 consensus_queue.cc:1035] T 01d518fafee34875a7df792d8297834b P 8b6a9ac9d6cd47b79f585d2981821b97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:00.532204 7494 consensus_queue.cc:1035] T 01d518fafee34875a7df792d8297834b P 8b6a9ac9d6cd47b79f585d2981821b97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:00.532444 7505 consensus_queue.cc:1035] T ef99459b1bb145689ba08b01e28e4093 P 8b6a9ac9d6cd47b79f585d2981821b97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:00.555971 7613 consensus_queue.cc:1035] T c1ce57f32584460f9bd94d85d19c8c4d P 8b6a9ac9d6cd47b79f585d2981821b97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:00.558748 7610 consensus_queue.cc:1035] T c1ce57f32584460f9bd94d85d19c8c4d P 8b6a9ac9d6cd47b79f585d2981821b97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:00.619146 7498 consensus_queue.cc:1035] T 42d98987ce3e4e82ad0e55e00cd49c24 P 97c52edf29a843bba2cd1653859f994e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:00.622216 7498 consensus_queue.cc:1035] T 42d98987ce3e4e82ad0e55e00cd49c24 P 97c52edf29a843bba2cd1653859f994e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:00.691742 7498 consensus_queue.cc:1035] T f3e2a90036a74f5a81d94a49595dc62a P 97c52edf29a843bba2cd1653859f994e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:00.695044 7620 consensus_queue.cc:1035] T f3e2a90036a74f5a81d94a49595dc62a P 97c52edf29a843bba2cd1653859f994e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:00.705230 7610 consensus_queue.cc:1035] T 7ba4aedf729d49bc977d5e52af888ebc P 8b6a9ac9d6cd47b79f585d2981821b97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:00.707715 7610 consensus_queue.cc:1035] T 7ba4aedf729d49bc977d5e52af888ebc P 8b6a9ac9d6cd47b79f585d2981821b97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:00.747115 7620 consensus_queue.cc:1035] T 7930116ca8d646faa009be917af87de8 P 97c52edf29a843bba2cd1653859f994e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:00.750072 7620 consensus_queue.cc:1035] T 7930116ca8d646faa009be917af87de8 P 97c52edf29a843bba2cd1653859f994e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:00.753201 7610 consensus_queue.cc:1035] T e5668a318ea44db8a7c4db7fc7c95f7a P 8b6a9ac9d6cd47b79f585d2981821b97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:00.755709 7610 consensus_queue.cc:1035] T e5668a318ea44db8a7c4db7fc7c95f7a P 8b6a9ac9d6cd47b79f585d2981821b97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:00.888741 7237 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:58:00.893962 7436 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7ba4aedf729d49bc977d5e52af888ebc"
dest_uuid: "8b6a9ac9d6cd47b79f585d2981821b97"
mode: GRACEFUL
new_leader_uuid: "5279c463ab5f4b6aa1d50c3fd0536174"
from {username='slave'} at 127.0.0.1:44586
I20240502 01:58:00.894147 7436 raft_consensus.cc:595] T 7ba4aedf729d49bc977d5e52af888ebc P 8b6a9ac9d6cd47b79f585d2981821b97 [term 1 LEADER]: Received request to transfer leadership to TS 5279c463ab5f4b6aa1d50c3fd0536174
I20240502 01:58:00.894937 7436 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c1ce57f32584460f9bd94d85d19c8c4d"
dest_uuid: "8b6a9ac9d6cd47b79f585d2981821b97"
mode: GRACEFUL
new_leader_uuid: "5279c463ab5f4b6aa1d50c3fd0536174"
from {username='slave'} at 127.0.0.1:44586
I20240502 01:58:00.895052 7436 raft_consensus.cc:595] T c1ce57f32584460f9bd94d85d19c8c4d P 8b6a9ac9d6cd47b79f585d2981821b97 [term 1 LEADER]: Received request to transfer leadership to TS 5279c463ab5f4b6aa1d50c3fd0536174
I20240502 01:58:00.895676 7436 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e5668a318ea44db8a7c4db7fc7c95f7a"
dest_uuid: "8b6a9ac9d6cd47b79f585d2981821b97"
mode: GRACEFUL
new_leader_uuid: "5279c463ab5f4b6aa1d50c3fd0536174"
from {username='slave'} at 127.0.0.1:44586
I20240502 01:58:00.895776 7436 raft_consensus.cc:595] T e5668a318ea44db8a7c4db7fc7c95f7a P 8b6a9ac9d6cd47b79f585d2981821b97 [term 1 LEADER]: Received request to transfer leadership to TS 5279c463ab5f4b6aa1d50c3fd0536174
I20240502 01:58:00.896052 7237 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 3
I20240502 01:58:00.896276 7237 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:58:01.167552 7613 raft_consensus.cc:979] T c1ce57f32584460f9bd94d85d19c8c4d P 8b6a9ac9d6cd47b79f585d2981821b97: : Instructing follower 5279c463ab5f4b6aa1d50c3fd0536174 to start an election
I20240502 01:58:01.167694 7613 raft_consensus.cc:1067] T c1ce57f32584460f9bd94d85d19c8c4d P 8b6a9ac9d6cd47b79f585d2981821b97 [term 1 LEADER]: Signalling peer 5279c463ab5f4b6aa1d50c3fd0536174 to start an election
I20240502 01:58:01.168298 7286 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c1ce57f32584460f9bd94d85d19c8c4d"
dest_uuid: "5279c463ab5f4b6aa1d50c3fd0536174"
from {username='slave'} at 127.0.0.1:54444
I20240502 01:58:01.168429 7286 raft_consensus.cc:482] T c1ce57f32584460f9bd94d85d19c8c4d P 5279c463ab5f4b6aa1d50c3fd0536174 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:58:01.168490 7286 raft_consensus.cc:3037] T c1ce57f32584460f9bd94d85d19c8c4d P 5279c463ab5f4b6aa1d50c3fd0536174 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:01.169261 7286 raft_consensus.cc:504] T c1ce57f32584460f9bd94d85d19c8c4d P 5279c463ab5f4b6aa1d50c3fd0536174 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } }
I20240502 01:58:01.169595 7286 leader_election.cc:288] T c1ce57f32584460f9bd94d85d19c8c4d P 5279c463ab5f4b6aa1d50c3fd0536174 [CANDIDATE]: Term 2 election: Requested vote from peers 97c52edf29a843bba2cd1653859f994e (127.6.117.194:44123), 8b6a9ac9d6cd47b79f585d2981821b97 (127.6.117.195:36729)
I20240502 01:58:01.169931 7361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1ce57f32584460f9bd94d85d19c8c4d" candidate_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "97c52edf29a843bba2cd1653859f994e"
I20240502 01:58:01.169909 7436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1ce57f32584460f9bd94d85d19c8c4d" candidate_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8b6a9ac9d6cd47b79f585d2981821b97"
I20240502 01:58:01.170063 7361 raft_consensus.cc:3037] T c1ce57f32584460f9bd94d85d19c8c4d P 97c52edf29a843bba2cd1653859f994e [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:01.170073 7436 raft_consensus.cc:3032] T c1ce57f32584460f9bd94d85d19c8c4d P 8b6a9ac9d6cd47b79f585d2981821b97 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:58:01.170125 7436 raft_consensus.cc:729] T c1ce57f32584460f9bd94d85d19c8c4d P 8b6a9ac9d6cd47b79f585d2981821b97 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8b6a9ac9d6cd47b79f585d2981821b97, State: Running, Role: LEADER
I20240502 01:58:01.170272 7436 consensus_queue.cc:260] T c1ce57f32584460f9bd94d85d19c8c4d P 8b6a9ac9d6cd47b79f585d2981821b97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } }
I20240502 01:58:01.170451 7436 raft_consensus.cc:3037] T c1ce57f32584460f9bd94d85d19c8c4d P 8b6a9ac9d6cd47b79f585d2981821b97 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:01.171010 7361 raft_consensus.cc:2454] T c1ce57f32584460f9bd94d85d19c8c4d P 97c52edf29a843bba2cd1653859f994e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5279c463ab5f4b6aa1d50c3fd0536174 in term 2.
I20240502 01:58:01.171288 7436 raft_consensus.cc:2454] T c1ce57f32584460f9bd94d85d19c8c4d P 8b6a9ac9d6cd47b79f585d2981821b97 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5279c463ab5f4b6aa1d50c3fd0536174 in term 2.
I20240502 01:58:01.171310 7249 leader_election.cc:302] T c1ce57f32584460f9bd94d85d19c8c4d P 5279c463ab5f4b6aa1d50c3fd0536174 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5279c463ab5f4b6aa1d50c3fd0536174, 97c52edf29a843bba2cd1653859f994e; no voters:
I20240502 01:58:01.171455 7489 raft_consensus.cc:2781] T c1ce57f32584460f9bd94d85d19c8c4d P 5279c463ab5f4b6aa1d50c3fd0536174 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:58:01.171635 7489 raft_consensus.cc:686] T c1ce57f32584460f9bd94d85d19c8c4d P 5279c463ab5f4b6aa1d50c3fd0536174 [term 2 LEADER]: Becoming Leader. State: Replica: 5279c463ab5f4b6aa1d50c3fd0536174, State: Running, Role: LEADER
I20240502 01:58:01.171769 7489 consensus_queue.cc:237] T c1ce57f32584460f9bd94d85d19c8c4d P 5279c463ab5f4b6aa1d50c3fd0536174 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } }
I20240502 01:58:01.173110 7183 catalog_manager.cc:5484] T c1ce57f32584460f9bd94d85d19c8c4d P 5279c463ab5f4b6aa1d50c3fd0536174 reported cstate change: term changed from 1 to 2, leader changed from 8b6a9ac9d6cd47b79f585d2981821b97 (127.6.117.195) to 5279c463ab5f4b6aa1d50c3fd0536174 (127.6.117.193). New cstate: current_term: 2 leader_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:01.262164 7610 raft_consensus.cc:979] T 7ba4aedf729d49bc977d5e52af888ebc P 8b6a9ac9d6cd47b79f585d2981821b97: : Instructing follower 5279c463ab5f4b6aa1d50c3fd0536174 to start an election
I20240502 01:58:01.262290 7610 raft_consensus.cc:1067] T 7ba4aedf729d49bc977d5e52af888ebc P 8b6a9ac9d6cd47b79f585d2981821b97 [term 1 LEADER]: Signalling peer 5279c463ab5f4b6aa1d50c3fd0536174 to start an election
I20240502 01:58:01.262712 7286 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7ba4aedf729d49bc977d5e52af888ebc"
dest_uuid: "5279c463ab5f4b6aa1d50c3fd0536174"
from {username='slave'} at 127.0.0.1:54444
I20240502 01:58:01.262856 7286 raft_consensus.cc:482] T 7ba4aedf729d49bc977d5e52af888ebc P 5279c463ab5f4b6aa1d50c3fd0536174 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:58:01.262943 7286 raft_consensus.cc:3037] T 7ba4aedf729d49bc977d5e52af888ebc P 5279c463ab5f4b6aa1d50c3fd0536174 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:01.263738 7286 raft_consensus.cc:504] T 7ba4aedf729d49bc977d5e52af888ebc P 5279c463ab5f4b6aa1d50c3fd0536174 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } }
I20240502 01:58:01.264102 7286 leader_election.cc:288] T 7ba4aedf729d49bc977d5e52af888ebc P 5279c463ab5f4b6aa1d50c3fd0536174 [CANDIDATE]: Term 2 election: Requested vote from peers 97c52edf29a843bba2cd1653859f994e (127.6.117.194:44123), 8b6a9ac9d6cd47b79f585d2981821b97 (127.6.117.195:36729)
I20240502 01:58:01.264410 7436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ba4aedf729d49bc977d5e52af888ebc" candidate_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8b6a9ac9d6cd47b79f585d2981821b97"
I20240502 01:58:01.264416 7361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ba4aedf729d49bc977d5e52af888ebc" candidate_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "97c52edf29a843bba2cd1653859f994e"
I20240502 01:58:01.264508 7436 raft_consensus.cc:3032] T 7ba4aedf729d49bc977d5e52af888ebc P 8b6a9ac9d6cd47b79f585d2981821b97 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:58:01.264593 7436 raft_consensus.cc:729] T 7ba4aedf729d49bc977d5e52af888ebc P 8b6a9ac9d6cd47b79f585d2981821b97 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8b6a9ac9d6cd47b79f585d2981821b97, State: Running, Role: LEADER
I20240502 01:58:01.264592 7361 raft_consensus.cc:3037] T 7ba4aedf729d49bc977d5e52af888ebc P 97c52edf29a843bba2cd1653859f994e [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:01.264742 7436 consensus_queue.cc:260] T 7ba4aedf729d49bc977d5e52af888ebc P 8b6a9ac9d6cd47b79f585d2981821b97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } }
I20240502 01:58:01.264884 7436 raft_consensus.cc:3037] T 7ba4aedf729d49bc977d5e52af888ebc P 8b6a9ac9d6cd47b79f585d2981821b97 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:01.265550 7361 raft_consensus.cc:2454] T 7ba4aedf729d49bc977d5e52af888ebc P 97c52edf29a843bba2cd1653859f994e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5279c463ab5f4b6aa1d50c3fd0536174 in term 2.
I20240502 01:58:01.265769 7436 raft_consensus.cc:2454] T 7ba4aedf729d49bc977d5e52af888ebc P 8b6a9ac9d6cd47b79f585d2981821b97 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5279c463ab5f4b6aa1d50c3fd0536174 in term 2.
I20240502 01:58:01.265830 7249 leader_election.cc:302] T 7ba4aedf729d49bc977d5e52af888ebc P 5279c463ab5f4b6aa1d50c3fd0536174 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5279c463ab5f4b6aa1d50c3fd0536174, 97c52edf29a843bba2cd1653859f994e; no voters:
I20240502 01:58:01.265982 7489 raft_consensus.cc:2781] T 7ba4aedf729d49bc977d5e52af888ebc P 5279c463ab5f4b6aa1d50c3fd0536174 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:58:01.266067 7489 raft_consensus.cc:686] T 7ba4aedf729d49bc977d5e52af888ebc P 5279c463ab5f4b6aa1d50c3fd0536174 [term 2 LEADER]: Becoming Leader. State: Replica: 5279c463ab5f4b6aa1d50c3fd0536174, State: Running, Role: LEADER
I20240502 01:58:01.266184 7489 consensus_queue.cc:237] T 7ba4aedf729d49bc977d5e52af888ebc P 5279c463ab5f4b6aa1d50c3fd0536174 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } }
I20240502 01:58:01.266242 7613 raft_consensus.cc:979] T e5668a318ea44db8a7c4db7fc7c95f7a P 8b6a9ac9d6cd47b79f585d2981821b97: : Instructing follower 5279c463ab5f4b6aa1d50c3fd0536174 to start an election
I20240502 01:58:01.266337 7613 raft_consensus.cc:1067] T e5668a318ea44db8a7c4db7fc7c95f7a P 8b6a9ac9d6cd47b79f585d2981821b97 [term 1 LEADER]: Signalling peer 5279c463ab5f4b6aa1d50c3fd0536174 to start an election
I20240502 01:58:01.266700 7286 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e5668a318ea44db8a7c4db7fc7c95f7a"
dest_uuid: "5279c463ab5f4b6aa1d50c3fd0536174"
from {username='slave'} at 127.0.0.1:54444
I20240502 01:58:01.266804 7286 raft_consensus.cc:482] T e5668a318ea44db8a7c4db7fc7c95f7a P 5279c463ab5f4b6aa1d50c3fd0536174 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:58:01.266858 7286 raft_consensus.cc:3037] T e5668a318ea44db8a7c4db7fc7c95f7a P 5279c463ab5f4b6aa1d50c3fd0536174 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:01.267621 7286 raft_consensus.cc:504] T e5668a318ea44db8a7c4db7fc7c95f7a P 5279c463ab5f4b6aa1d50c3fd0536174 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } }
I20240502 01:58:01.267585 7183 catalog_manager.cc:5484] T 7ba4aedf729d49bc977d5e52af888ebc P 5279c463ab5f4b6aa1d50c3fd0536174 reported cstate change: term changed from 1 to 2, leader changed from 8b6a9ac9d6cd47b79f585d2981821b97 (127.6.117.195) to 5279c463ab5f4b6aa1d50c3fd0536174 (127.6.117.193). New cstate: current_term: 2 leader_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:01.267926 7286 leader_election.cc:288] T e5668a318ea44db8a7c4db7fc7c95f7a P 5279c463ab5f4b6aa1d50c3fd0536174 [CANDIDATE]: Term 2 election: Requested vote from peers 97c52edf29a843bba2cd1653859f994e (127.6.117.194:44123), 8b6a9ac9d6cd47b79f585d2981821b97 (127.6.117.195:36729)
I20240502 01:58:01.268200 7361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5668a318ea44db8a7c4db7fc7c95f7a" candidate_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "97c52edf29a843bba2cd1653859f994e"
I20240502 01:58:01.268339 7436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5668a318ea44db8a7c4db7fc7c95f7a" candidate_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8b6a9ac9d6cd47b79f585d2981821b97"
I20240502 01:58:01.268349 7361 raft_consensus.cc:3037] T e5668a318ea44db8a7c4db7fc7c95f7a P 97c52edf29a843bba2cd1653859f994e [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:01.268463 7436 raft_consensus.cc:3032] T e5668a318ea44db8a7c4db7fc7c95f7a P 8b6a9ac9d6cd47b79f585d2981821b97 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:58:01.268505 7436 raft_consensus.cc:729] T e5668a318ea44db8a7c4db7fc7c95f7a P 8b6a9ac9d6cd47b79f585d2981821b97 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8b6a9ac9d6cd47b79f585d2981821b97, State: Running, Role: LEADER
I20240502 01:58:01.268627 7436 consensus_queue.cc:260] T e5668a318ea44db8a7c4db7fc7c95f7a P 8b6a9ac9d6cd47b79f585d2981821b97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } }
I20240502 01:58:01.268776 7436 raft_consensus.cc:3037] T e5668a318ea44db8a7c4db7fc7c95f7a P 8b6a9ac9d6cd47b79f585d2981821b97 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:01.269217 7361 raft_consensus.cc:2454] T e5668a318ea44db8a7c4db7fc7c95f7a P 97c52edf29a843bba2cd1653859f994e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5279c463ab5f4b6aa1d50c3fd0536174 in term 2.
I20240502 01:58:01.269485 7249 leader_election.cc:302] T e5668a318ea44db8a7c4db7fc7c95f7a P 5279c463ab5f4b6aa1d50c3fd0536174 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5279c463ab5f4b6aa1d50c3fd0536174, 97c52edf29a843bba2cd1653859f994e; no voters:
I20240502 01:58:01.269492 7436 raft_consensus.cc:2454] T e5668a318ea44db8a7c4db7fc7c95f7a P 8b6a9ac9d6cd47b79f585d2981821b97 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5279c463ab5f4b6aa1d50c3fd0536174 in term 2.
I20240502 01:58:01.269699 7489 raft_consensus.cc:2781] T e5668a318ea44db8a7c4db7fc7c95f7a P 5279c463ab5f4b6aa1d50c3fd0536174 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:58:01.269771 7489 raft_consensus.cc:686] T e5668a318ea44db8a7c4db7fc7c95f7a P 5279c463ab5f4b6aa1d50c3fd0536174 [term 2 LEADER]: Becoming Leader. State: Replica: 5279c463ab5f4b6aa1d50c3fd0536174, State: Running, Role: LEADER
I20240502 01:58:01.269872 7489 consensus_queue.cc:237] T e5668a318ea44db8a7c4db7fc7c95f7a P 5279c463ab5f4b6aa1d50c3fd0536174 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } }
I20240502 01:58:01.272055 7183 catalog_manager.cc:5484] T e5668a318ea44db8a7c4db7fc7c95f7a P 5279c463ab5f4b6aa1d50c3fd0536174 reported cstate change: term changed from 1 to 2, leader changed from 8b6a9ac9d6cd47b79f585d2981821b97 (127.6.117.195) to 5279c463ab5f4b6aa1d50c3fd0536174 (127.6.117.193). New cstate: current_term: 2 leader_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:01.439529 7604 heartbeater.cc:503] Master 127.6.117.254:35435 was elected leader, sending a full tablet report...
I20240502 01:58:01.700438 7361 raft_consensus.cc:1261] T e5668a318ea44db8a7c4db7fc7c95f7a P 97c52edf29a843bba2cd1653859f994e [term 2 FOLLOWER]: Refusing update from remote peer 5279c463ab5f4b6aa1d50c3fd0536174: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 01:58:01.700932 7632 consensus_queue.cc:1035] T e5668a318ea44db8a7c4db7fc7c95f7a P 5279c463ab5f4b6aa1d50c3fd0536174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:58:01.703011 7436 raft_consensus.cc:1261] T e5668a318ea44db8a7c4db7fc7c95f7a P 8b6a9ac9d6cd47b79f585d2981821b97 [term 2 FOLLOWER]: Refusing update from remote peer 5279c463ab5f4b6aa1d50c3fd0536174: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 01:58:01.703357 7489 consensus_queue.cc:1035] T e5668a318ea44db8a7c4db7fc7c95f7a P 5279c463ab5f4b6aa1d50c3fd0536174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:58:01.747617 7361 raft_consensus.cc:1261] T c1ce57f32584460f9bd94d85d19c8c4d P 97c52edf29a843bba2cd1653859f994e [term 2 FOLLOWER]: Refusing update from remote peer 5279c463ab5f4b6aa1d50c3fd0536174: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 01:58:01.748080 7637 consensus_queue.cc:1035] T c1ce57f32584460f9bd94d85d19c8c4d P 5279c463ab5f4b6aa1d50c3fd0536174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:58:01.750299 7436 raft_consensus.cc:1261] T c1ce57f32584460f9bd94d85d19c8c4d P 8b6a9ac9d6cd47b79f585d2981821b97 [term 2 FOLLOWER]: Refusing update from remote peer 5279c463ab5f4b6aa1d50c3fd0536174: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 01:58:01.750625 7489 consensus_queue.cc:1035] T c1ce57f32584460f9bd94d85d19c8c4d P 5279c463ab5f4b6aa1d50c3fd0536174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:58:01.812103 7361 raft_consensus.cc:1261] T 7ba4aedf729d49bc977d5e52af888ebc P 97c52edf29a843bba2cd1653859f994e [term 2 FOLLOWER]: Refusing update from remote peer 5279c463ab5f4b6aa1d50c3fd0536174: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 01:58:01.812533 7489 consensus_queue.cc:1035] T 7ba4aedf729d49bc977d5e52af888ebc P 5279c463ab5f4b6aa1d50c3fd0536174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:58:01.815886 7436 raft_consensus.cc:1261] T 7ba4aedf729d49bc977d5e52af888ebc P 8b6a9ac9d6cd47b79f585d2981821b97 [term 2 FOLLOWER]: Refusing update from remote peer 5279c463ab5f4b6aa1d50c3fd0536174: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 01:58:01.816288 7632 consensus_queue.cc:1035] T 7ba4aedf729d49bc977d5e52af888ebc P 5279c463ab5f4b6aa1d50c3fd0536174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:58:02.896521 7237 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:58:02.900908 7286 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e5668a318ea44db8a7c4db7fc7c95f7a"
dest_uuid: "5279c463ab5f4b6aa1d50c3fd0536174"
mode: GRACEFUL
new_leader_uuid: "8b6a9ac9d6cd47b79f585d2981821b97"
from {username='slave'} at 127.0.0.1:54464
I20240502 01:58:02.901052 7286 raft_consensus.cc:595] T e5668a318ea44db8a7c4db7fc7c95f7a P 5279c463ab5f4b6aa1d50c3fd0536174 [term 2 LEADER]: Received request to transfer leadership to TS 8b6a9ac9d6cd47b79f585d2981821b97
I20240502 01:58:02.901381 7237 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240502 01:58:02.901495 7237 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:58:03.213910 7637 raft_consensus.cc:979] T e5668a318ea44db8a7c4db7fc7c95f7a P 5279c463ab5f4b6aa1d50c3fd0536174: : Instructing follower 8b6a9ac9d6cd47b79f585d2981821b97 to start an election
I20240502 01:58:03.214021 7637 raft_consensus.cc:1067] T e5668a318ea44db8a7c4db7fc7c95f7a P 5279c463ab5f4b6aa1d50c3fd0536174 [term 2 LEADER]: Signalling peer 8b6a9ac9d6cd47b79f585d2981821b97 to start an election
I20240502 01:58:03.214452 7436 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e5668a318ea44db8a7c4db7fc7c95f7a"
dest_uuid: "8b6a9ac9d6cd47b79f585d2981821b97"
from {username='slave'} at 127.0.0.1:44584
I20240502 01:58:03.214607 7436 raft_consensus.cc:482] T e5668a318ea44db8a7c4db7fc7c95f7a P 8b6a9ac9d6cd47b79f585d2981821b97 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:58:03.214679 7436 raft_consensus.cc:3037] T e5668a318ea44db8a7c4db7fc7c95f7a P 8b6a9ac9d6cd47b79f585d2981821b97 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:58:03.215605 7436 raft_consensus.cc:504] T e5668a318ea44db8a7c4db7fc7c95f7a P 8b6a9ac9d6cd47b79f585d2981821b97 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } }
I20240502 01:58:03.215929 7436 leader_election.cc:288] T e5668a318ea44db8a7c4db7fc7c95f7a P 8b6a9ac9d6cd47b79f585d2981821b97 [CANDIDATE]: Term 3 election: Requested vote from peers 5279c463ab5f4b6aa1d50c3fd0536174 (127.6.117.193:44887), 97c52edf29a843bba2cd1653859f994e (127.6.117.194:44123)
I20240502 01:58:03.216229 7286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5668a318ea44db8a7c4db7fc7c95f7a" candidate_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "5279c463ab5f4b6aa1d50c3fd0536174"
I20240502 01:58:03.216289 7361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5668a318ea44db8a7c4db7fc7c95f7a" candidate_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "97c52edf29a843bba2cd1653859f994e"
I20240502 01:58:03.216363 7286 raft_consensus.cc:3032] T e5668a318ea44db8a7c4db7fc7c95f7a P 5279c463ab5f4b6aa1d50c3fd0536174 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:58:03.216399 7361 raft_consensus.cc:3037] T e5668a318ea44db8a7c4db7fc7c95f7a P 97c52edf29a843bba2cd1653859f994e [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:58:03.216403 7286 raft_consensus.cc:729] T e5668a318ea44db8a7c4db7fc7c95f7a P 5279c463ab5f4b6aa1d50c3fd0536174 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5279c463ab5f4b6aa1d50c3fd0536174, State: Running, Role: LEADER
I20240502 01:58:03.216638 7286 consensus_queue.cc:260] T e5668a318ea44db8a7c4db7fc7c95f7a P 5279c463ab5f4b6aa1d50c3fd0536174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } }
I20240502 01:58:03.216786 7286 raft_consensus.cc:3037] T e5668a318ea44db8a7c4db7fc7c95f7a P 5279c463ab5f4b6aa1d50c3fd0536174 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:58:03.217228 7361 raft_consensus.cc:2454] T e5668a318ea44db8a7c4db7fc7c95f7a P 97c52edf29a843bba2cd1653859f994e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b6a9ac9d6cd47b79f585d2981821b97 in term 3.
I20240502 01:58:03.217479 7286 raft_consensus.cc:2454] T e5668a318ea44db8a7c4db7fc7c95f7a P 5279c463ab5f4b6aa1d50c3fd0536174 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b6a9ac9d6cd47b79f585d2981821b97 in term 3.
I20240502 01:58:03.217497 7400 leader_election.cc:302] T e5668a318ea44db8a7c4db7fc7c95f7a P 8b6a9ac9d6cd47b79f585d2981821b97 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b6a9ac9d6cd47b79f585d2981821b97, 97c52edf29a843bba2cd1653859f994e; no voters:
I20240502 01:58:03.217635 7613 raft_consensus.cc:2781] T e5668a318ea44db8a7c4db7fc7c95f7a P 8b6a9ac9d6cd47b79f585d2981821b97 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:58:03.217833 7613 raft_consensus.cc:686] T e5668a318ea44db8a7c4db7fc7c95f7a P 8b6a9ac9d6cd47b79f585d2981821b97 [term 3 LEADER]: Becoming Leader. State: Replica: 8b6a9ac9d6cd47b79f585d2981821b97, State: Running, Role: LEADER
I20240502 01:58:03.217978 7613 consensus_queue.cc:237] T e5668a318ea44db8a7c4db7fc7c95f7a P 8b6a9ac9d6cd47b79f585d2981821b97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } }
I20240502 01:58:03.219347 7183 catalog_manager.cc:5484] T e5668a318ea44db8a7c4db7fc7c95f7a P 8b6a9ac9d6cd47b79f585d2981821b97 reported cstate change: term changed from 2 to 3, leader changed from 5279c463ab5f4b6aa1d50c3fd0536174 (127.6.117.193) to 8b6a9ac9d6cd47b79f585d2981821b97 (127.6.117.195). New cstate: current_term: 3 leader_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } health_report { overall_health: HEALTHY } } }
I20240502 01:58:03.701980 7286 raft_consensus.cc:1261] T e5668a318ea44db8a7c4db7fc7c95f7a P 5279c463ab5f4b6aa1d50c3fd0536174 [term 3 FOLLOWER]: Refusing update from remote peer 8b6a9ac9d6cd47b79f585d2981821b97: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 01:58:03.702457 7613 consensus_queue.cc:1035] T e5668a318ea44db8a7c4db7fc7c95f7a P 8b6a9ac9d6cd47b79f585d2981821b97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:58:03.705119 7361 raft_consensus.cc:1261] T e5668a318ea44db8a7c4db7fc7c95f7a P 97c52edf29a843bba2cd1653859f994e [term 3 FOLLOWER]: Refusing update from remote peer 8b6a9ac9d6cd47b79f585d2981821b97: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 01:58:03.705770 7654 consensus_queue.cc:1035] T e5668a318ea44db8a7c4db7fc7c95f7a P 8b6a9ac9d6cd47b79f585d2981821b97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:58:04.901827 7237 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:58:04.903395 7436 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "01d518fafee34875a7df792d8297834b"
dest_uuid: "8b6a9ac9d6cd47b79f585d2981821b97"
mode: GRACEFUL
new_leader_uuid: "5279c463ab5f4b6aa1d50c3fd0536174"
from {username='slave'} at 127.0.0.1:44586
I20240502 01:58:04.903559 7436 raft_consensus.cc:595] T 01d518fafee34875a7df792d8297834b P 8b6a9ac9d6cd47b79f585d2981821b97 [term 1 LEADER]: Received request to transfer leadership to TS 5279c463ab5f4b6aa1d50c3fd0536174
I20240502 01:58:04.903918 7237 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240502 01:58:04.904027 7237 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:58:05.066715 7646 raft_consensus.cc:979] T 01d518fafee34875a7df792d8297834b P 8b6a9ac9d6cd47b79f585d2981821b97: : Instructing follower 5279c463ab5f4b6aa1d50c3fd0536174 to start an election
I20240502 01:58:05.066829 7646 raft_consensus.cc:1067] T 01d518fafee34875a7df792d8297834b P 8b6a9ac9d6cd47b79f585d2981821b97 [term 1 LEADER]: Signalling peer 5279c463ab5f4b6aa1d50c3fd0536174 to start an election
I20240502 01:58:05.067329 7286 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "01d518fafee34875a7df792d8297834b"
dest_uuid: "5279c463ab5f4b6aa1d50c3fd0536174"
from {username='slave'} at 127.0.0.1:54444
I20240502 01:58:05.067472 7286 raft_consensus.cc:482] T 01d518fafee34875a7df792d8297834b P 5279c463ab5f4b6aa1d50c3fd0536174 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:58:05.067536 7286 raft_consensus.cc:3037] T 01d518fafee34875a7df792d8297834b P 5279c463ab5f4b6aa1d50c3fd0536174 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:05.068528 7286 raft_consensus.cc:504] T 01d518fafee34875a7df792d8297834b P 5279c463ab5f4b6aa1d50c3fd0536174 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } } peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } }
I20240502 01:58:05.068912 7286 leader_election.cc:288] T 01d518fafee34875a7df792d8297834b P 5279c463ab5f4b6aa1d50c3fd0536174 [CANDIDATE]: Term 2 election: Requested vote from peers 8b6a9ac9d6cd47b79f585d2981821b97 (127.6.117.195:36729), 97c52edf29a843bba2cd1653859f994e (127.6.117.194:44123)
I20240502 01:58:05.069221 7436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01d518fafee34875a7df792d8297834b" candidate_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8b6a9ac9d6cd47b79f585d2981821b97"
I20240502 01:58:05.069283 7361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01d518fafee34875a7df792d8297834b" candidate_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "97c52edf29a843bba2cd1653859f994e"
I20240502 01:58:05.069351 7436 raft_consensus.cc:3032] T 01d518fafee34875a7df792d8297834b P 8b6a9ac9d6cd47b79f585d2981821b97 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:58:05.069351 7361 raft_consensus.cc:3037] T 01d518fafee34875a7df792d8297834b P 97c52edf29a843bba2cd1653859f994e [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:05.069433 7436 raft_consensus.cc:729] T 01d518fafee34875a7df792d8297834b P 8b6a9ac9d6cd47b79f585d2981821b97 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8b6a9ac9d6cd47b79f585d2981821b97, State: Running, Role: LEADER
I20240502 01:58:05.069586 7436 consensus_queue.cc:260] T 01d518fafee34875a7df792d8297834b P 8b6a9ac9d6cd47b79f585d2981821b97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } } peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } }
I20240502 01:58:05.069753 7436 raft_consensus.cc:3037] T 01d518fafee34875a7df792d8297834b P 8b6a9ac9d6cd47b79f585d2981821b97 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:05.070384 7361 raft_consensus.cc:2454] T 01d518fafee34875a7df792d8297834b P 97c52edf29a843bba2cd1653859f994e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5279c463ab5f4b6aa1d50c3fd0536174 in term 2.
I20240502 01:58:05.070577 7436 raft_consensus.cc:2454] T 01d518fafee34875a7df792d8297834b P 8b6a9ac9d6cd47b79f585d2981821b97 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5279c463ab5f4b6aa1d50c3fd0536174 in term 2.
I20240502 01:58:05.070705 7249 leader_election.cc:302] T 01d518fafee34875a7df792d8297834b P 5279c463ab5f4b6aa1d50c3fd0536174 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5279c463ab5f4b6aa1d50c3fd0536174, 97c52edf29a843bba2cd1653859f994e; no voters:
I20240502 01:58:05.070927 7637 raft_consensus.cc:2781] T 01d518fafee34875a7df792d8297834b P 5279c463ab5f4b6aa1d50c3fd0536174 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:58:05.071132 7637 raft_consensus.cc:686] T 01d518fafee34875a7df792d8297834b P 5279c463ab5f4b6aa1d50c3fd0536174 [term 2 LEADER]: Becoming Leader. State: Replica: 5279c463ab5f4b6aa1d50c3fd0536174, State: Running, Role: LEADER
I20240502 01:58:05.071272 7637 consensus_queue.cc:237] T 01d518fafee34875a7df792d8297834b P 5279c463ab5f4b6aa1d50c3fd0536174 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } } peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } }
I20240502 01:58:05.072799 7183 catalog_manager.cc:5484] T 01d518fafee34875a7df792d8297834b P 5279c463ab5f4b6aa1d50c3fd0536174 reported cstate change: term changed from 1 to 2, leader changed from 8b6a9ac9d6cd47b79f585d2981821b97 (127.6.117.195) to 5279c463ab5f4b6aa1d50c3fd0536174 (127.6.117.193). New cstate: current_term: 2 leader_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:05.508605 7361 raft_consensus.cc:1261] T 01d518fafee34875a7df792d8297834b P 97c52edf29a843bba2cd1653859f994e [term 2 FOLLOWER]: Refusing update from remote peer 5279c463ab5f4b6aa1d50c3fd0536174: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 01:58:05.509024 7637 consensus_queue.cc:1035] T 01d518fafee34875a7df792d8297834b P 5279c463ab5f4b6aa1d50c3fd0536174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:58:05.511456 7436 raft_consensus.cc:1261] T 01d518fafee34875a7df792d8297834b P 8b6a9ac9d6cd47b79f585d2981821b97 [term 2 FOLLOWER]: Refusing update from remote peer 5279c463ab5f4b6aa1d50c3fd0536174: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 01:58:05.512001 7668 consensus_queue.cc:1035] T 01d518fafee34875a7df792d8297834b P 5279c463ab5f4b6aa1d50c3fd0536174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:58:05.902222 7361 consensus_queue.cc:237] T 42d98987ce3e4e82ad0e55e00cd49c24 P 97c52edf29a843bba2cd1653859f994e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } attrs { replace: true } } peers { permanent_uuid: "a1c5c6906f434cea9d1966623e1ff641" member_type: NON_VOTER last_known_addr { host: "127.6.117.196" port: 34745 } attrs { promote: true } }
I20240502 01:58:05.903820 7286 raft_consensus.cc:1261] T 42d98987ce3e4e82ad0e55e00cd49c24 P 5279c463ab5f4b6aa1d50c3fd0536174 [term 1 FOLLOWER]: Refusing update from remote peer 97c52edf29a843bba2cd1653859f994e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:05.904053 7436 raft_consensus.cc:1261] T 42d98987ce3e4e82ad0e55e00cd49c24 P 8b6a9ac9d6cd47b79f585d2981821b97 [term 1 FOLLOWER]: Refusing update from remote peer 97c52edf29a843bba2cd1653859f994e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:05.904203 7676 consensus_queue.cc:1035] T 42d98987ce3e4e82ad0e55e00cd49c24 P 97c52edf29a843bba2cd1653859f994e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:58:05.904448 7675 consensus_queue.cc:1035] T 42d98987ce3e4e82ad0e55e00cd49c24 P 97c52edf29a843bba2cd1653859f994e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:58:05.906145 7665 raft_consensus.cc:2932] T 42d98987ce3e4e82ad0e55e00cd49c24 P 97c52edf29a843bba2cd1653859f994e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a1c5c6906f434cea9d1966623e1ff641 (127.6.117.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } attrs { replace: true } } peers { permanent_uuid: "a1c5c6906f434cea9d1966623e1ff641" member_type: NON_VOTER last_known_addr { host: "127.6.117.196" port: 34745 } attrs { promote: true } } }
I20240502 01:58:05.906584 7286 raft_consensus.cc:2932] T 42d98987ce3e4e82ad0e55e00cd49c24 P 5279c463ab5f4b6aa1d50c3fd0536174 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a1c5c6906f434cea9d1966623e1ff641 (127.6.117.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } attrs { replace: true } } peers { permanent_uuid: "a1c5c6906f434cea9d1966623e1ff641" member_type: NON_VOTER last_known_addr { host: "127.6.117.196" port: 34745 } attrs { promote: true } } }
I20240502 01:58:05.907590 7436 raft_consensus.cc:2932] T 42d98987ce3e4e82ad0e55e00cd49c24 P 8b6a9ac9d6cd47b79f585d2981821b97 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a1c5c6906f434cea9d1966623e1ff641 (127.6.117.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } attrs { replace: true } } peers { permanent_uuid: "a1c5c6906f434cea9d1966623e1ff641" member_type: NON_VOTER last_known_addr { host: "127.6.117.196" port: 34745 } attrs { promote: true } } }
W20240502 01:58:05.907830 7328 consensus_peers.cc:485] T 42d98987ce3e4e82ad0e55e00cd49c24 P 97c52edf29a843bba2cd1653859f994e -> Peer a1c5c6906f434cea9d1966623e1ff641 (127.6.117.196:34745): Couldn't send request to peer a1c5c6906f434cea9d1966623e1ff641. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 42d98987ce3e4e82ad0e55e00cd49c24. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:05.961439 7181 catalog_manager.cc:5484] T 42d98987ce3e4e82ad0e55e00cd49c24 P 8b6a9ac9d6cd47b79f585d2981821b97 reported cstate change: config changed from index -1 to 2, NON_VOTER a1c5c6906f434cea9d1966623e1ff641 (127.6.117.196) added. New cstate: current_term: 1 leader_uuid: "97c52edf29a843bba2cd1653859f994e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } attrs { replace: true } } peers { permanent_uuid: "a1c5c6906f434cea9d1966623e1ff641" member_type: NON_VOTER last_known_addr { host: "127.6.117.196" port: 34745 } attrs { promote: true } } }
I20240502 01:58:05.966840 7286 consensus_queue.cc:237] T 01d518fafee34875a7df792d8297834b P 5279c463ab5f4b6aa1d50c3fd0536174 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } } peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "a1c5c6906f434cea9d1966623e1ff641" member_type: NON_VOTER last_known_addr { host: "127.6.117.196" port: 34745 } attrs { promote: true } }
I20240502 01:58:05.968111 7361 raft_consensus.cc:1261] T 01d518fafee34875a7df792d8297834b P 97c52edf29a843bba2cd1653859f994e [term 2 FOLLOWER]: Refusing update from remote peer 5279c463ab5f4b6aa1d50c3fd0536174: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 01:58:05.968307 7436 raft_consensus.cc:1261] T 01d518fafee34875a7df792d8297834b P 8b6a9ac9d6cd47b79f585d2981821b97 [term 2 FOLLOWER]: Refusing update from remote peer 5279c463ab5f4b6aa1d50c3fd0536174: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 01:58:05.968644 7659 consensus_queue.cc:1035] T 01d518fafee34875a7df792d8297834b P 5279c463ab5f4b6aa1d50c3fd0536174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:58:05.968802 7668 consensus_queue.cc:1035] T 01d518fafee34875a7df792d8297834b P 5279c463ab5f4b6aa1d50c3fd0536174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:58:05.970201 7637 raft_consensus.cc:2932] T 01d518fafee34875a7df792d8297834b P 5279c463ab5f4b6aa1d50c3fd0536174 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER a1c5c6906f434cea9d1966623e1ff641 (127.6.117.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } } peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "a1c5c6906f434cea9d1966623e1ff641" member_type: NON_VOTER last_known_addr { host: "127.6.117.196" port: 34745 } attrs { promote: true } } }
I20240502 01:58:05.971446 7361 raft_consensus.cc:2932] T 01d518fafee34875a7df792d8297834b P 97c52edf29a843bba2cd1653859f994e [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER a1c5c6906f434cea9d1966623e1ff641 (127.6.117.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } } peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "a1c5c6906f434cea9d1966623e1ff641" member_type: NON_VOTER last_known_addr { host: "127.6.117.196" port: 34745 } attrs { promote: true } } }
I20240502 01:58:05.971493 7436 raft_consensus.cc:2932] T 01d518fafee34875a7df792d8297834b P 8b6a9ac9d6cd47b79f585d2981821b97 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER a1c5c6906f434cea9d1966623e1ff641 (127.6.117.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } } peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "a1c5c6906f434cea9d1966623e1ff641" member_type: NON_VOTER last_known_addr { host: "127.6.117.196" port: 34745 } attrs { promote: true } } }
W20240502 01:58:05.972221 7252 consensus_peers.cc:485] T 01d518fafee34875a7df792d8297834b P 5279c463ab5f4b6aa1d50c3fd0536174 -> Peer a1c5c6906f434cea9d1966623e1ff641 (127.6.117.196:34745): Couldn't send request to peer a1c5c6906f434cea9d1966623e1ff641. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 01d518fafee34875a7df792d8297834b. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:06.064049 7183 catalog_manager.cc:5484] T 01d518fafee34875a7df792d8297834b P 97c52edf29a843bba2cd1653859f994e reported cstate change: config changed from index -1 to 3, NON_VOTER a1c5c6906f434cea9d1966623e1ff641 (127.6.117.196) added. New cstate: current_term: 2 leader_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } } peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "a1c5c6906f434cea9d1966623e1ff641" member_type: NON_VOTER last_known_addr { host: "127.6.117.196" port: 34745 } attrs { promote: true } } }
I20240502 01:58:06.069034 7436 consensus_queue.cc:237] T e5668a318ea44db8a7c4db7fc7c95f7a P 8b6a9ac9d6cd47b79f585d2981821b97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } attrs { replace: true } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } } peers { permanent_uuid: "a1c5c6906f434cea9d1966623e1ff641" member_type: NON_VOTER last_known_addr { host: "127.6.117.196" port: 34745 } attrs { promote: true } }
I20240502 01:58:06.070457 7286 raft_consensus.cc:1261] T e5668a318ea44db8a7c4db7fc7c95f7a P 5279c463ab5f4b6aa1d50c3fd0536174 [term 3 FOLLOWER]: Refusing update from remote peer 8b6a9ac9d6cd47b79f585d2981821b97: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240502 01:58:06.070487 7361 raft_consensus.cc:1261] T e5668a318ea44db8a7c4db7fc7c95f7a P 97c52edf29a843bba2cd1653859f994e [term 3 FOLLOWER]: Refusing update from remote peer 8b6a9ac9d6cd47b79f585d2981821b97: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240502 01:58:06.070819 7689 consensus_queue.cc:1035] T e5668a318ea44db8a7c4db7fc7c95f7a P 8b6a9ac9d6cd47b79f585d2981821b97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:06.071023 7646 consensus_queue.cc:1035] T e5668a318ea44db8a7c4db7fc7c95f7a P 8b6a9ac9d6cd47b79f585d2981821b97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:06.072652 7689 raft_consensus.cc:2932] T e5668a318ea44db8a7c4db7fc7c95f7a P 8b6a9ac9d6cd47b79f585d2981821b97 [term 3 LEADER]: Committing config change with OpId 3.4: config changed from index -1 to 4, NON_VOTER a1c5c6906f434cea9d1966623e1ff641 (127.6.117.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } attrs { replace: true } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } } peers { permanent_uuid: "a1c5c6906f434cea9d1966623e1ff641" member_type: NON_VOTER last_known_addr { host: "127.6.117.196" port: 34745 } attrs { promote: true } } }
I20240502 01:58:06.073097 7361 raft_consensus.cc:2932] T e5668a318ea44db8a7c4db7fc7c95f7a P 97c52edf29a843bba2cd1653859f994e [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index -1 to 4, NON_VOTER a1c5c6906f434cea9d1966623e1ff641 (127.6.117.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } attrs { replace: true } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } } peers { permanent_uuid: "a1c5c6906f434cea9d1966623e1ff641" member_type: NON_VOTER last_known_addr { host: "127.6.117.196" port: 34745 } attrs { promote: true } } }
I20240502 01:58:06.073534 7286 raft_consensus.cc:2932] T e5668a318ea44db8a7c4db7fc7c95f7a P 5279c463ab5f4b6aa1d50c3fd0536174 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index -1 to 4, NON_VOTER a1c5c6906f434cea9d1966623e1ff641 (127.6.117.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } attrs { replace: true } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } } peers { permanent_uuid: "a1c5c6906f434cea9d1966623e1ff641" member_type: NON_VOTER last_known_addr { host: "127.6.117.196" port: 34745 } attrs { promote: true } } }
W20240502 01:58:06.075259 7403 consensus_peers.cc:485] T e5668a318ea44db8a7c4db7fc7c95f7a P 8b6a9ac9d6cd47b79f585d2981821b97 -> Peer a1c5c6906f434cea9d1966623e1ff641 (127.6.117.196:34745): Couldn't send request to peer a1c5c6906f434cea9d1966623e1ff641. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e5668a318ea44db8a7c4db7fc7c95f7a. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:06.167327 7183 catalog_manager.cc:5484] T e5668a318ea44db8a7c4db7fc7c95f7a P 8b6a9ac9d6cd47b79f585d2981821b97 reported cstate change: config changed from index -1 to 4, NON_VOTER a1c5c6906f434cea9d1966623e1ff641 (127.6.117.196) added. New cstate: current_term: 3 leader_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1c5c6906f434cea9d1966623e1ff641" member_type: NON_VOTER last_known_addr { host: "127.6.117.196" port: 34745 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:06.167594 7286 consensus_queue.cc:237] T c1ce57f32584460f9bd94d85d19c8c4d P 5279c463ab5f4b6aa1d50c3fd0536174 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } } peers { permanent_uuid: "a1c5c6906f434cea9d1966623e1ff641" member_type: NON_VOTER last_known_addr { host: "127.6.117.196" port: 34745 } attrs { promote: true } }
I20240502 01:58:06.169250 7436 raft_consensus.cc:1261] T c1ce57f32584460f9bd94d85d19c8c4d P 8b6a9ac9d6cd47b79f585d2981821b97 [term 2 FOLLOWER]: Refusing update from remote peer 5279c463ab5f4b6aa1d50c3fd0536174: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
W20240502 01:58:06.169395 7252 consensus_peers.cc:485] T c1ce57f32584460f9bd94d85d19c8c4d P 5279c463ab5f4b6aa1d50c3fd0536174 -> Peer a1c5c6906f434cea9d1966623e1ff641 (127.6.117.196:34745): Couldn't send request to peer a1c5c6906f434cea9d1966623e1ff641. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c1ce57f32584460f9bd94d85d19c8c4d. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:06.169567 7637 consensus_queue.cc:1035] T c1ce57f32584460f9bd94d85d19c8c4d P 5279c463ab5f4b6aa1d50c3fd0536174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:58:06.170246 7361 raft_consensus.cc:1261] T c1ce57f32584460f9bd94d85d19c8c4d P 97c52edf29a843bba2cd1653859f994e [term 2 FOLLOWER]: Refusing update from remote peer 5279c463ab5f4b6aa1d50c3fd0536174: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 01:58:06.170639 7637 consensus_queue.cc:1035] T c1ce57f32584460f9bd94d85d19c8c4d P 5279c463ab5f4b6aa1d50c3fd0536174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:58:06.171361 7659 raft_consensus.cc:2932] T c1ce57f32584460f9bd94d85d19c8c4d P 5279c463ab5f4b6aa1d50c3fd0536174 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER a1c5c6906f434cea9d1966623e1ff641 (127.6.117.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } } peers { permanent_uuid: "a1c5c6906f434cea9d1966623e1ff641" member_type: NON_VOTER last_known_addr { host: "127.6.117.196" port: 34745 } attrs { promote: true } } }
I20240502 01:58:06.171742 7436 raft_consensus.cc:2932] T c1ce57f32584460f9bd94d85d19c8c4d P 8b6a9ac9d6cd47b79f585d2981821b97 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER a1c5c6906f434cea9d1966623e1ff641 (127.6.117.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } } peers { permanent_uuid: "a1c5c6906f434cea9d1966623e1ff641" member_type: NON_VOTER last_known_addr { host: "127.6.117.196" port: 34745 } attrs { promote: true } } }
I20240502 01:58:06.172551 7361 raft_consensus.cc:2932] T c1ce57f32584460f9bd94d85d19c8c4d P 97c52edf29a843bba2cd1653859f994e [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER a1c5c6906f434cea9d1966623e1ff641 (127.6.117.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } } peers { permanent_uuid: "a1c5c6906f434cea9d1966623e1ff641" member_type: NON_VOTER last_known_addr { host: "127.6.117.196" port: 34745 } attrs { promote: true } } }
I20240502 01:58:06.263208 7182 catalog_manager.cc:5484] T c1ce57f32584460f9bd94d85d19c8c4d P 97c52edf29a843bba2cd1653859f994e reported cstate change: config changed from index -1 to 3, NON_VOTER a1c5c6906f434cea9d1966623e1ff641 (127.6.117.196) added. New cstate: current_term: 2 leader_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } } peers { permanent_uuid: "a1c5c6906f434cea9d1966623e1ff641" member_type: NON_VOTER last_known_addr { host: "127.6.117.196" port: 34745 } attrs { promote: true } } }
I20240502 01:58:06.400835 7696 ts_tablet_manager.cc:902] T 01d518fafee34875a7df792d8297834b P a1c5c6906f434cea9d1966623e1ff641: Initiating tablet copy from peer 5279c463ab5f4b6aa1d50c3fd0536174 (127.6.117.193:44887)
I20240502 01:58:06.401710 7696 tablet_copy_client.cc:308] T 01d518fafee34875a7df792d8297834b P a1c5c6906f434cea9d1966623e1ff641: tablet copy: Beginning tablet copy session from remote peer at address 127.6.117.193:44887
I20240502 01:58:06.405463 7296 tablet_copy_service.cc:140] P 5279c463ab5f4b6aa1d50c3fd0536174: Received BeginTabletCopySession request for tablet 01d518fafee34875a7df792d8297834b from peer a1c5c6906f434cea9d1966623e1ff641 ({username='slave'} at 127.0.0.1:54488)
I20240502 01:58:06.405599 7296 tablet_copy_service.cc:161] P 5279c463ab5f4b6aa1d50c3fd0536174: Beginning new tablet copy session on tablet 01d518fafee34875a7df792d8297834b from peer a1c5c6906f434cea9d1966623e1ff641 at {username='slave'} at 127.0.0.1:54488: session id = a1c5c6906f434cea9d1966623e1ff641-01d518fafee34875a7df792d8297834b
I20240502 01:58:06.406754 7296 tablet_copy_source_session.cc:205] T 01d518fafee34875a7df792d8297834b P 5279c463ab5f4b6aa1d50c3fd0536174: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:06.408195 7696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01d518fafee34875a7df792d8297834b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:06.410441 7696 tablet_copy_client.cc:783] T 01d518fafee34875a7df792d8297834b P a1c5c6906f434cea9d1966623e1ff641: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:06.410714 7702 ts_tablet_manager.cc:902] T 42d98987ce3e4e82ad0e55e00cd49c24 P a1c5c6906f434cea9d1966623e1ff641: Initiating tablet copy from peer 97c52edf29a843bba2cd1653859f994e (127.6.117.194:44123)
I20240502 01:58:06.410714 7696 tablet_copy_client.cc:647] T 01d518fafee34875a7df792d8297834b P a1c5c6906f434cea9d1966623e1ff641: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:06.411434 7702 tablet_copy_client.cc:308] T 42d98987ce3e4e82ad0e55e00cd49c24 P a1c5c6906f434cea9d1966623e1ff641: tablet copy: Beginning tablet copy session from remote peer at address 127.6.117.194:44123
I20240502 01:58:06.412441 7696 tablet_copy_client.cc:523] T 01d518fafee34875a7df792d8297834b P a1c5c6906f434cea9d1966623e1ff641: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:06.415449 7372 tablet_copy_service.cc:140] P 97c52edf29a843bba2cd1653859f994e: Received BeginTabletCopySession request for tablet 42d98987ce3e4e82ad0e55e00cd49c24 from peer a1c5c6906f434cea9d1966623e1ff641 ({username='slave'} at 127.0.0.1:50590)
I20240502 01:58:06.415542 7372 tablet_copy_service.cc:161] P 97c52edf29a843bba2cd1653859f994e: Beginning new tablet copy session on tablet 42d98987ce3e4e82ad0e55e00cd49c24 from peer a1c5c6906f434cea9d1966623e1ff641 at {username='slave'} at 127.0.0.1:50590: session id = a1c5c6906f434cea9d1966623e1ff641-42d98987ce3e4e82ad0e55e00cd49c24
I20240502 01:58:06.416347 7372 tablet_copy_source_session.cc:205] T 42d98987ce3e4e82ad0e55e00cd49c24 P 97c52edf29a843bba2cd1653859f994e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:06.461766 7702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42d98987ce3e4e82ad0e55e00cd49c24. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:06.462375 7696 tablet_bootstrap.cc:492] T 01d518fafee34875a7df792d8297834b P a1c5c6906f434cea9d1966623e1ff641: Bootstrap starting.
I20240502 01:58:06.464128 7702 tablet_copy_client.cc:783] T 42d98987ce3e4e82ad0e55e00cd49c24 P a1c5c6906f434cea9d1966623e1ff641: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:06.464304 7702 tablet_copy_client.cc:647] T 42d98987ce3e4e82ad0e55e00cd49c24 P a1c5c6906f434cea9d1966623e1ff641: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:06.464735 6615 tablet_server.cc:178] TabletServer@127.6.117.193:0 shutting down...
I20240502 01:58:06.465447 7702 tablet_copy_client.cc:523] T 42d98987ce3e4e82ad0e55e00cd49c24 P a1c5c6906f434cea9d1966623e1ff641: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:06.466982 6615 tablet_copy_service.cc:362] P 5279c463ab5f4b6aa1d50c3fd0536174: Destroying tablet copy session a1c5c6906f434cea9d1966623e1ff641-01d518fafee34875a7df792d8297834b due to service shutdown
W20240502 01:58:06.467023 7400 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 1 similar messages]
I20240502 01:58:06.467051 6615 tablet_copy_service.cc:433] P 5279c463ab5f4b6aa1d50c3fd0536174: ending tablet copy session a1c5c6906f434cea9d1966623e1ff641-01d518fafee34875a7df792d8297834b on tablet 01d518fafee34875a7df792d8297834b with peer a1c5c6906f434cea9d1966623e1ff641
I20240502 01:58:06.468051 7696 tablet_bootstrap.cc:492] T 01d518fafee34875a7df792d8297834b P a1c5c6906f434cea9d1966623e1ff641: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:06.468919 7696 tablet_bootstrap.cc:492] T 01d518fafee34875a7df792d8297834b P a1c5c6906f434cea9d1966623e1ff641: Bootstrap complete.
I20240502 01:58:06.469877 7696 ts_tablet_manager.cc:1367] T 01d518fafee34875a7df792d8297834b P a1c5c6906f434cea9d1966623e1ff641: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
W20240502 01:58:06.470314 7400 consensus_peers.cc:485] T e5668a318ea44db8a7c4db7fc7c95f7a P 8b6a9ac9d6cd47b79f585d2981821b97 -> Peer 5279c463ab5f4b6aa1d50c3fd0536174 (127.6.117.193:44887): Couldn't send request to peer 5279c463ab5f4b6aa1d50c3fd0536174. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:06.470433 6615 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:58:06.470584 6615 tablet_replica.cc:330] T ef99459b1bb145689ba08b01e28e4093 P 5279c463ab5f4b6aa1d50c3fd0536174: stopping tablet replica
I20240502 01:58:06.470541 7696 raft_consensus.cc:348] T 01d518fafee34875a7df792d8297834b P a1c5c6906f434cea9d1966623e1ff641 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } } peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "a1c5c6906f434cea9d1966623e1ff641" member_type: NON_VOTER last_known_addr { host: "127.6.117.196" port: 34745 } attrs { promote: true } }
I20240502 01:58:06.470664 7696 raft_consensus.cc:729] T 01d518fafee34875a7df792d8297834b P a1c5c6906f434cea9d1966623e1ff641 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: a1c5c6906f434cea9d1966623e1ff641, State: Initialized, Role: LEARNER
I20240502 01:58:06.470712 6615 raft_consensus.cc:2229] T ef99459b1bb145689ba08b01e28e4093 P 5279c463ab5f4b6aa1d50c3fd0536174 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:06.470808 6615 raft_consensus.cc:2258] T ef99459b1bb145689ba08b01e28e4093 P 5279c463ab5f4b6aa1d50c3fd0536174 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:06.470790 7696 consensus_queue.cc:260] T 01d518fafee34875a7df792d8297834b P a1c5c6906f434cea9d1966623e1ff641 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } } peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "a1c5c6906f434cea9d1966623e1ff641" member_type: NON_VOTER last_known_addr { host: "127.6.117.196" port: 34745 } attrs { promote: true } }
I20240502 01:58:06.471107 7696 ts_tablet_manager.cc:1398] T 01d518fafee34875a7df792d8297834b P a1c5c6906f434cea9d1966623e1ff641: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:06.471190 6615 tablet_replica.cc:330] T 42d98987ce3e4e82ad0e55e00cd49c24 P 5279c463ab5f4b6aa1d50c3fd0536174: stopping tablet replica
I20240502 01:58:06.471283 6615 raft_consensus.cc:2229] T 42d98987ce3e4e82ad0e55e00cd49c24 P 5279c463ab5f4b6aa1d50c3fd0536174 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:06.471441 6615 raft_consensus.cc:2258] T 42d98987ce3e4e82ad0e55e00cd49c24 P 5279c463ab5f4b6aa1d50c3fd0536174 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240502 01:58:06.471838 7231 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240502 01:58:06.471889 7231 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240502 01:58:06.472054 6615 tablet_replica.cc:330] T 7ba4aedf729d49bc977d5e52af888ebc P 5279c463ab5f4b6aa1d50c3fd0536174: stopping tablet replica
I20240502 01:58:06.472126 6615 raft_consensus.cc:2229] T 7ba4aedf729d49bc977d5e52af888ebc P 5279c463ab5f4b6aa1d50c3fd0536174 [term 2 LEADER]: Raft consensus shutting down.
I20240502 01:58:06.472294 6615 raft_consensus.cc:2258] T 7ba4aedf729d49bc977d5e52af888ebc P 5279c463ab5f4b6aa1d50c3fd0536174 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:06.472940 6615 tablet_replica.cc:330] T 01d518fafee34875a7df792d8297834b P 5279c463ab5f4b6aa1d50c3fd0536174: stopping tablet replica
I20240502 01:58:06.473071 6615 raft_consensus.cc:2229] T 01d518fafee34875a7df792d8297834b P 5279c463ab5f4b6aa1d50c3fd0536174 [term 2 LEADER]: Raft consensus shutting down.
I20240502 01:58:06.473264 6615 raft_consensus.cc:2258] T 01d518fafee34875a7df792d8297834b P 5279c463ab5f4b6aa1d50c3fd0536174 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:06.473676 6615 tablet_replica.cc:330] T c1ce57f32584460f9bd94d85d19c8c4d P 5279c463ab5f4b6aa1d50c3fd0536174: stopping tablet replica
I20240502 01:58:06.473763 6615 raft_consensus.cc:2229] T c1ce57f32584460f9bd94d85d19c8c4d P 5279c463ab5f4b6aa1d50c3fd0536174 [term 2 LEADER]: Raft consensus shutting down.
I20240502 01:58:06.473977 6615 raft_consensus.cc:2258] T c1ce57f32584460f9bd94d85d19c8c4d P 5279c463ab5f4b6aa1d50c3fd0536174 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:06.474476 6615 tablet_replica.cc:330] T f3e2a90036a74f5a81d94a49595dc62a P 5279c463ab5f4b6aa1d50c3fd0536174: stopping tablet replica
I20240502 01:58:06.474553 6615 raft_consensus.cc:2229] T f3e2a90036a74f5a81d94a49595dc62a P 5279c463ab5f4b6aa1d50c3fd0536174 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:06.474637 6615 raft_consensus.cc:2258] T f3e2a90036a74f5a81d94a49595dc62a P 5279c463ab5f4b6aa1d50c3fd0536174 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240502 01:58:06.474869 7231 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240502 01:58:06.474922 7231 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240502 01:58:06.474944 6615 tablet_replica.cc:330] T e5668a318ea44db8a7c4db7fc7c95f7a P 5279c463ab5f4b6aa1d50c3fd0536174: stopping tablet replica
I20240502 01:58:06.475028 6615 raft_consensus.cc:2229] T e5668a318ea44db8a7c4db7fc7c95f7a P 5279c463ab5f4b6aa1d50c3fd0536174 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:06.475118 6615 raft_consensus.cc:2258] T e5668a318ea44db8a7c4db7fc7c95f7a P 5279c463ab5f4b6aa1d50c3fd0536174 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:06.475545 6615 tablet_replica.cc:330] T 7930116ca8d646faa009be917af87de8 P 5279c463ab5f4b6aa1d50c3fd0536174: stopping tablet replica
I20240502 01:58:06.475625 6615 raft_consensus.cc:2229] T 7930116ca8d646faa009be917af87de8 P 5279c463ab5f4b6aa1d50c3fd0536174 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:06.475708 6615 raft_consensus.cc:2258] T 7930116ca8d646faa009be917af87de8 P 5279c463ab5f4b6aa1d50c3fd0536174 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240502 01:58:06.478288 7696 tablet_copy_client.cc:1107] T 01d518fafee34875a7df792d8297834b P a1c5c6906f434cea9d1966623e1ff641: tablet copy: Unable to close tablet copy session: Network error: failure ending tablet copy session: Client connection negotiation failed: client connection to 127.6.117.193:44887: connect: Connection refused (error 111)
I20240502 01:58:06.490696 6615 tablet_server.cc:194] TabletServer@127.6.117.193:0 shutdown complete.
I20240502 01:58:06.493531 6615 tablet_server.cc:178] TabletServer@127.6.117.194:0 shutting down...
I20240502 01:58:06.495239 6615 tablet_copy_service.cc:362] P 97c52edf29a843bba2cd1653859f994e: Destroying tablet copy session a1c5c6906f434cea9d1966623e1ff641-42d98987ce3e4e82ad0e55e00cd49c24 due to service shutdown
I20240502 01:58:06.495347 6615 tablet_copy_service.cc:433] P 97c52edf29a843bba2cd1653859f994e: ending tablet copy session a1c5c6906f434cea9d1966623e1ff641-42d98987ce3e4e82ad0e55e00cd49c24 on tablet 42d98987ce3e4e82ad0e55e00cd49c24 with peer a1c5c6906f434cea9d1966623e1ff641
W20240502 01:58:06.496311 7231 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240502 01:58:06.496373 7231 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240502 01:58:06.498586 6615 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:58:06.498730 6615 tablet_replica.cc:330] T 01d518fafee34875a7df792d8297834b P 97c52edf29a843bba2cd1653859f994e: stopping tablet replica
I20240502 01:58:06.498884 6615 raft_consensus.cc:2229] T 01d518fafee34875a7df792d8297834b P 97c52edf29a843bba2cd1653859f994e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:06.499059 6615 raft_consensus.cc:2258] T 01d518fafee34875a7df792d8297834b P 97c52edf29a843bba2cd1653859f994e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:06.499313 7696 ts_tablet_manager.cc:902] T e5668a318ea44db8a7c4db7fc7c95f7a P a1c5c6906f434cea9d1966623e1ff641: Initiating tablet copy from peer 8b6a9ac9d6cd47b79f585d2981821b97 (127.6.117.195:36729)
I20240502 01:58:06.499759 7696 tablet_copy_client.cc:308] T e5668a318ea44db8a7c4db7fc7c95f7a P a1c5c6906f434cea9d1966623e1ff641: tablet copy: Beginning tablet copy session from remote peer at address 127.6.117.195:36729
I20240502 01:58:06.499770 6615 tablet_replica.cc:330] T e5668a318ea44db8a7c4db7fc7c95f7a P 97c52edf29a843bba2cd1653859f994e: stopping tablet replica
I20240502 01:58:06.499922 6615 raft_consensus.cc:2229] T e5668a318ea44db8a7c4db7fc7c95f7a P 97c52edf29a843bba2cd1653859f994e [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:06.500069 6615 raft_consensus.cc:2258] T e5668a318ea44db8a7c4db7fc7c95f7a P 97c52edf29a843bba2cd1653859f994e [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:06.500648 6615 tablet_replica.cc:330] T f3e2a90036a74f5a81d94a49595dc62a P 97c52edf29a843bba2cd1653859f994e: stopping tablet replica
I20240502 01:58:06.500790 6615 raft_consensus.cc:2229] T f3e2a90036a74f5a81d94a49595dc62a P 97c52edf29a843bba2cd1653859f994e [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:58:06.500993 6615 raft_consensus.cc:2258] T f3e2a90036a74f5a81d94a49595dc62a P 97c52edf29a843bba2cd1653859f994e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:06.501380 6615 tablet_replica.cc:330] T 42d98987ce3e4e82ad0e55e00cd49c24 P 97c52edf29a843bba2cd1653859f994e: stopping tablet replica
I20240502 01:58:06.501471 6615 raft_consensus.cc:2229] T 42d98987ce3e4e82ad0e55e00cd49c24 P 97c52edf29a843bba2cd1653859f994e [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:58:06.501680 6615 raft_consensus.cc:2258] T 42d98987ce3e4e82ad0e55e00cd49c24 P 97c52edf29a843bba2cd1653859f994e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:06.502117 6615 tablet_replica.cc:330] T ef99459b1bb145689ba08b01e28e4093 P 97c52edf29a843bba2cd1653859f994e: stopping tablet replica
I20240502 01:58:06.502204 6615 raft_consensus.cc:2229] T ef99459b1bb145689ba08b01e28e4093 P 97c52edf29a843bba2cd1653859f994e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:06.502290 6615 raft_consensus.cc:2258] T ef99459b1bb145689ba08b01e28e4093 P 97c52edf29a843bba2cd1653859f994e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:06.502583 6615 tablet_replica.cc:330] T 7930116ca8d646faa009be917af87de8 P 97c52edf29a843bba2cd1653859f994e: stopping tablet replica
I20240502 01:58:06.502662 6615 raft_consensus.cc:2229] T 7930116ca8d646faa009be917af87de8 P 97c52edf29a843bba2cd1653859f994e [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:58:06.502805 6615 raft_consensus.cc:2258] T 7930116ca8d646faa009be917af87de8 P 97c52edf29a843bba2cd1653859f994e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:06.503194 6615 tablet_replica.cc:330] T 7ba4aedf729d49bc977d5e52af888ebc P 97c52edf29a843bba2cd1653859f994e: stopping tablet replica
I20240502 01:58:06.503286 6615 raft_consensus.cc:2229] T 7ba4aedf729d49bc977d5e52af888ebc P 97c52edf29a843bba2cd1653859f994e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:06.503378 6615 raft_consensus.cc:2258] T 7ba4aedf729d49bc977d5e52af888ebc P 97c52edf29a843bba2cd1653859f994e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:06.503741 6615 tablet_replica.cc:330] T c1ce57f32584460f9bd94d85d19c8c4d P 97c52edf29a843bba2cd1653859f994e: stopping tablet replica
I20240502 01:58:06.503775 7447 tablet_copy_service.cc:140] P 8b6a9ac9d6cd47b79f585d2981821b97: Received BeginTabletCopySession request for tablet e5668a318ea44db8a7c4db7fc7c95f7a from peer a1c5c6906f434cea9d1966623e1ff641 ({username='slave'} at 127.0.0.1:44614)
I20240502 01:58:06.503819 6615 raft_consensus.cc:2229] T c1ce57f32584460f9bd94d85d19c8c4d P 97c52edf29a843bba2cd1653859f994e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:06.503890 7447 tablet_copy_service.cc:161] P 8b6a9ac9d6cd47b79f585d2981821b97: Beginning new tablet copy session on tablet e5668a318ea44db8a7c4db7fc7c95f7a from peer a1c5c6906f434cea9d1966623e1ff641 at {username='slave'} at 127.0.0.1:44614: session id = a1c5c6906f434cea9d1966623e1ff641-e5668a318ea44db8a7c4db7fc7c95f7a
I20240502 01:58:06.503929 6615 raft_consensus.cc:2258] T c1ce57f32584460f9bd94d85d19c8c4d P 97c52edf29a843bba2cd1653859f994e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:06.504725 7447 tablet_copy_source_session.cc:205] T e5668a318ea44db8a7c4db7fc7c95f7a P 8b6a9ac9d6cd47b79f585d2981821b97: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:06.518662 6615 tablet_server.cc:194] TabletServer@127.6.117.194:0 shutdown complete.
I20240502 01:58:06.521780 6615 tablet_server.cc:178] TabletServer@127.6.117.195:0 shutting down...
I20240502 01:58:06.523450 6615 tablet_copy_service.cc:362] P 8b6a9ac9d6cd47b79f585d2981821b97: Destroying tablet copy session a1c5c6906f434cea9d1966623e1ff641-e5668a318ea44db8a7c4db7fc7c95f7a due to service shutdown
I20240502 01:58:06.523519 6615 tablet_copy_service.cc:433] P 8b6a9ac9d6cd47b79f585d2981821b97: ending tablet copy session a1c5c6906f434cea9d1966623e1ff641-e5668a318ea44db8a7c4db7fc7c95f7a on tablet e5668a318ea44db8a7c4db7fc7c95f7a with peer a1c5c6906f434cea9d1966623e1ff641
W20240502 01:58:06.524021 7231 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240502 01:58:06.524077 7231 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240502 01:58:06.526567 6615 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:58:06.526707 6615 tablet_replica.cc:330] T 7930116ca8d646faa009be917af87de8 P 8b6a9ac9d6cd47b79f585d2981821b97: stopping tablet replica
I20240502 01:58:06.526839 6615 raft_consensus.cc:2229] T 7930116ca8d646faa009be917af87de8 P 8b6a9ac9d6cd47b79f585d2981821b97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:06.526942 6615 raft_consensus.cc:2258] T 7930116ca8d646faa009be917af87de8 P 8b6a9ac9d6cd47b79f585d2981821b97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:06.527303 6615 tablet_replica.cc:330] T 42d98987ce3e4e82ad0e55e00cd49c24 P 8b6a9ac9d6cd47b79f585d2981821b97: stopping tablet replica
I20240502 01:58:06.527385 6615 raft_consensus.cc:2229] T 42d98987ce3e4e82ad0e55e00cd49c24 P 8b6a9ac9d6cd47b79f585d2981821b97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:06.527467 6615 raft_consensus.cc:2258] T 42d98987ce3e4e82ad0e55e00cd49c24 P 8b6a9ac9d6cd47b79f585d2981821b97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:06.527848 6615 tablet_replica.cc:330] T e5668a318ea44db8a7c4db7fc7c95f7a P 8b6a9ac9d6cd47b79f585d2981821b97: stopping tablet replica
I20240502 01:58:06.527923 6615 raft_consensus.cc:2229] T e5668a318ea44db8a7c4db7fc7c95f7a P 8b6a9ac9d6cd47b79f585d2981821b97 [term 3 LEADER]: Raft consensus shutting down.
I20240502 01:58:06.528065 6615 raft_consensus.cc:2258] T e5668a318ea44db8a7c4db7fc7c95f7a P 8b6a9ac9d6cd47b79f585d2981821b97 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:06.528455 6615 tablet_replica.cc:330] T f3e2a90036a74f5a81d94a49595dc62a P 8b6a9ac9d6cd47b79f585d2981821b97: stopping tablet replica
I20240502 01:58:06.528534 6615 raft_consensus.cc:2229] T f3e2a90036a74f5a81d94a49595dc62a P 8b6a9ac9d6cd47b79f585d2981821b97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:06.528602 6615 raft_consensus.cc:2258] T f3e2a90036a74f5a81d94a49595dc62a P 8b6a9ac9d6cd47b79f585d2981821b97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:06.528846 6615 tablet_replica.cc:330] T 01d518fafee34875a7df792d8297834b P 8b6a9ac9d6cd47b79f585d2981821b97: stopping tablet replica
I20240502 01:58:06.528913 6615 raft_consensus.cc:2229] T 01d518fafee34875a7df792d8297834b P 8b6a9ac9d6cd47b79f585d2981821b97 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:06.528983 6615 raft_consensus.cc:2258] T 01d518fafee34875a7df792d8297834b P 8b6a9ac9d6cd47b79f585d2981821b97 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:06.529345 6615 tablet_replica.cc:330] T c1ce57f32584460f9bd94d85d19c8c4d P 8b6a9ac9d6cd47b79f585d2981821b97: stopping tablet replica
I20240502 01:58:06.529417 6615 raft_consensus.cc:2229] T c1ce57f32584460f9bd94d85d19c8c4d P 8b6a9ac9d6cd47b79f585d2981821b97 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:06.529498 6615 raft_consensus.cc:2258] T c1ce57f32584460f9bd94d85d19c8c4d P 8b6a9ac9d6cd47b79f585d2981821b97 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:06.529908 6615 tablet_replica.cc:330] T ef99459b1bb145689ba08b01e28e4093 P 8b6a9ac9d6cd47b79f585d2981821b97: stopping tablet replica
I20240502 01:58:06.529975 6615 raft_consensus.cc:2229] T ef99459b1bb145689ba08b01e28e4093 P 8b6a9ac9d6cd47b79f585d2981821b97 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:58:06.530100 6615 raft_consensus.cc:2258] T ef99459b1bb145689ba08b01e28e4093 P 8b6a9ac9d6cd47b79f585d2981821b97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:06.530349 6615 tablet_replica.cc:330] T 7ba4aedf729d49bc977d5e52af888ebc P 8b6a9ac9d6cd47b79f585d2981821b97: stopping tablet replica
I20240502 01:58:06.530411 6615 raft_consensus.cc:2229] T 7ba4aedf729d49bc977d5e52af888ebc P 8b6a9ac9d6cd47b79f585d2981821b97 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:06.530475 6615 raft_consensus.cc:2258] T 7ba4aedf729d49bc977d5e52af888ebc P 8b6a9ac9d6cd47b79f585d2981821b97 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:06.533717 6615 tablet_server.cc:194] TabletServer@127.6.117.195:0 shutdown complete.
I20240502 01:58:06.536173 6615 tablet_server.cc:178] TabletServer@127.6.117.196:0 shutting down...
I20240502 01:58:06.539934 6615 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:58:06.565567 7696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5668a318ea44db8a7c4db7fc7c95f7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:06.566192 7702 tablet_bootstrap.cc:492] T 42d98987ce3e4e82ad0e55e00cd49c24 P a1c5c6906f434cea9d1966623e1ff641: Bootstrap starting.
I20240502 01:58:06.568197 7696 tablet_copy_client.cc:783] T e5668a318ea44db8a7c4db7fc7c95f7a P a1c5c6906f434cea9d1966623e1ff641: tablet copy: Starting download of 0 data blocks...
W20240502 01:58:06.568344 7541 connection.cc:441] server connection from 127.0.0.1:37320 torn down before Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:37320 (request call id 1) could send its response
I20240502 01:58:06.568356 7696 tablet_copy_client.cc:647] T e5668a318ea44db8a7c4db7fc7c95f7a P a1c5c6906f434cea9d1966623e1ff641: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:06.569953 7702 tablet_bootstrap.cc:492] T 42d98987ce3e4e82ad0e55e00cd49c24 P a1c5c6906f434cea9d1966623e1ff641: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
W20240502 01:58:06.569981 7696 ts_tablet_manager.cc:980] T e5668a318ea44db8a7c4db7fc7c95f7a P a1c5c6906f434cea9d1966623e1ff641: Tablet Copy: Unable to fetch data from remote peer 8b6a9ac9d6cd47b79f585d2981821b97 (127.6.117.195:36729): Network error: Unable to download WAL segment with seq. number 1: unable to fetch data from remote: Client connection negotiation failed: client connection to 127.6.117.195:36729: connect: Connection refused (error 111)
I20240502 01:58:06.570056 7696 tablet_replica.cc:330] T e5668a318ea44db8a7c4db7fc7c95f7a P a1c5c6906f434cea9d1966623e1ff641: stopping tablet replica
I20240502 01:58:06.570101 7696 raft_consensus.cc:2229] T e5668a318ea44db8a7c4db7fc7c95f7a P a1c5c6906f434cea9d1966623e1ff641 [term 3 LEARNER]: Raft consensus shutting down.
I20240502 01:58:06.570155 7696 raft_consensus.cc:2258] T e5668a318ea44db8a7c4db7fc7c95f7a P a1c5c6906f434cea9d1966623e1ff641 [term 3 LEARNER]: Raft consensus is shut down!
I20240502 01:58:06.570189 7702 tablet_bootstrap.cc:492] T 42d98987ce3e4e82ad0e55e00cd49c24 P a1c5c6906f434cea9d1966623e1ff641: Bootstrap complete.
I20240502 01:58:06.570362 7702 ts_tablet_manager.cc:1367] T 42d98987ce3e4e82ad0e55e00cd49c24 P a1c5c6906f434cea9d1966623e1ff641: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
W20240502 01:58:06.570768 7696 tablet_copy_client.cc:1107] T e5668a318ea44db8a7c4db7fc7c95f7a P a1c5c6906f434cea9d1966623e1ff641: tablet copy: Unable to close tablet copy session: Network error: failure ending tablet copy session: Client connection negotiation failed: client connection to 127.6.117.195:36729: connect: Connection refused (error 111)
I20240502 01:58:06.570794 7702 raft_consensus.cc:348] T 42d98987ce3e4e82ad0e55e00cd49c24 P a1c5c6906f434cea9d1966623e1ff641 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } attrs { replace: true } } peers { permanent_uuid: "a1c5c6906f434cea9d1966623e1ff641" member_type: NON_VOTER last_known_addr { host: "127.6.117.196" port: 34745 } attrs { promote: true } }
I20240502 01:58:06.570930 7702 raft_consensus.cc:729] T 42d98987ce3e4e82ad0e55e00cd49c24 P a1c5c6906f434cea9d1966623e1ff641 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a1c5c6906f434cea9d1966623e1ff641, State: Initialized, Role: LEARNER
I20240502 01:58:06.571036 7702 consensus_queue.cc:260] T 42d98987ce3e4e82ad0e55e00cd49c24 P a1c5c6906f434cea9d1966623e1ff641 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5279c463ab5f4b6aa1d50c3fd0536174" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 44887 } } peers { permanent_uuid: "97c52edf29a843bba2cd1653859f994e" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 44123 } } peers { permanent_uuid: "8b6a9ac9d6cd47b79f585d2981821b97" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 36729 } attrs { replace: true } } peers { permanent_uuid: "a1c5c6906f434cea9d1966623e1ff641" member_type: NON_VOTER last_known_addr { host: "127.6.117.196" port: 34745 } attrs { promote: true } }
W20240502 01:58:06.571063 7696 tablet_metadata.cc:474] failed to load DataDirGroup from superblock: Already present: tried to load directory group for tablet e5668a318ea44db8a7c4db7fc7c95f7a but one is already registered
I20240502 01:58:06.571192 7696 ts_tablet_manager.cc:1875] T e5668a318ea44db8a7c4db7fc7c95f7a P a1c5c6906f434cea9d1966623e1ff641: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:06.571281 7702 ts_tablet_manager.cc:1398] T 42d98987ce3e4e82ad0e55e00cd49c24 P a1c5c6906f434cea9d1966623e1ff641: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20240502 01:58:06.571805 7702 tablet_copy_client.cc:1107] T 42d98987ce3e4e82ad0e55e00cd49c24 P a1c5c6906f434cea9d1966623e1ff641: tablet copy: Unable to close tablet copy session: Network error: failure ending tablet copy session: Client connection negotiation failed: client connection to 127.6.117.194:44123: connect: Connection refused (error 111)
I20240502 01:58:06.667956 7696 ts_tablet_manager.cc:1888] T e5668a318ea44db8a7c4db7fc7c95f7a P a1c5c6906f434cea9d1966623e1ff641: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.0
I20240502 01:58:06.668071 7696 log.cc:1196] T e5668a318ea44db8a7c4db7fc7c95f7a P a1c5c6906f434cea9d1966623e1ff641: Deleting WAL directory at /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714615076189642-6615-0/minicluster-data/ts-3-root/wals/e5668a318ea44db8a7c4db7fc7c95f7a
I20240502 01:58:06.668897 6615 tablet_replica.cc:330] T 01d518fafee34875a7df792d8297834b P a1c5c6906f434cea9d1966623e1ff641: stopping tablet replica
I20240502 01:58:06.669066 6615 raft_consensus.cc:2229] T 01d518fafee34875a7df792d8297834b P a1c5c6906f434cea9d1966623e1ff641 [term 2 LEARNER]: Raft consensus shutting down.
I20240502 01:58:06.669184 6615 raft_consensus.cc:2258] T 01d518fafee34875a7df792d8297834b P a1c5c6906f434cea9d1966623e1ff641 [term 2 LEARNER]: Raft consensus is shut down!
I20240502 01:58:06.669653 6615 tablet_replica.cc:330] T 42d98987ce3e4e82ad0e55e00cd49c24 P a1c5c6906f434cea9d1966623e1ff641: stopping tablet replica
I20240502 01:58:06.669739 6615 raft_consensus.cc:2229] T 42d98987ce3e4e82ad0e55e00cd49c24 P a1c5c6906f434cea9d1966623e1ff641 [term 1 LEARNER]: Raft consensus shutting down.
I20240502 01:58:06.669835 6615 raft_consensus.cc:2258] T 42d98987ce3e4e82ad0e55e00cd49c24 P a1c5c6906f434cea9d1966623e1ff641 [term 1 LEARNER]: Raft consensus is shut down!
I20240502 01:58:06.682452 6615 tablet_server.cc:194] TabletServer@127.6.117.196:0 shutdown complete.
I20240502 01:58:06.684602 6615 master.cc:537] Master@127.6.117.254:35435 shutting down...
I20240502 01:58:06.688199 6615 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 2bd74cf1295c44a1950e5eee7b5199fe [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:58:06.688452 6615 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 2bd74cf1295c44a1950e5eee7b5199fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:06.688529 6615 tablet_replica.cc:330] T 00000000000000000000000000000000 P 2bd74cf1295c44a1950e5eee7b5199fe: stopping tablet replica
I20240502 01:58:06.701185 6615 master.cc:552] Master@127.6.117.254:35435 shutdown complete.
[ OK ] AutoRebalancerTest.AutoRebalancingTurnOffAndOn (7855 ms)
[ RUN ] AutoRebalancerTest.NextLeaderResumesAutoRebalancing
I20240502 01:58:06.713512 6615 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.6.117.254:44289,127.6.117.253:39267,127.6.117.252:44737
I20240502 01:58:06.713925 6615 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:58:06.716053 7718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:06.716493 7717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:06.716825 7720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:58:06.717118 6615 server_base.cc:1034] running on GCE node
I20240502 01:58:06.717252 6615 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:58:06.717286 6615 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:58:06.717305 6615 hybrid_clock.cc:648] HybridClock initialized: now 1714615086717305 us; error 0 us; skew 500 ppm
I20240502 01:58:06.718166 6615 webserver.cc:460] Webserver started at http://127.6.117.254:36075/ using document root <none> and password file <none>
I20240502 01:58:06.718312 6615 fs_manager.cc:359] Metadata directory not provided
I20240502 01:58:06.718361 6615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:58:06.718442 6615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:58:06.718811 6615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714615076189642-6615-0/minicluster-data/master-0-root/instance:
uuid: "ba1379406ff942b883b7fca5a80c5873"
format_stamp: "Formatted at 2024-05-02 01:58:06 on dist-test-slave-89cm"
I20240502 01:58:06.720155 6615 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:06.721061 7725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:06.721315 6615 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240502 01:58:06.721377 6615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714615076189642-6615-0/minicluster-data/master-0-root
uuid: "ba1379406ff942b883b7fca5a80c5873"
format_stamp: "Formatted at 2024-05-02 01:58:06 on dist-test-slave-89cm"
I20240502 01:58:06.721434 6615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714615076189642-6615-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714615076189642-6615-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714615076189642-6615-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:58:06.739063 6615 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:58:06.739408 6615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:58:06.743249 6615 rpc_server.cc:307] RPC server started. Bound to: 127.6.117.254:44289
I20240502 01:58:06.744378 6615 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 01:58:06.747961 7777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.117.254:44289 every 8 connection(s)
I20240502 01:58:06.748520 7778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 01:58:06.749163 7783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:06.749501 7781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:06.749886 7780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:58:06.749857 7778 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.117.254" port: 44289 } has no permanent_uuid. Determining permanent_uuid...
I20240502 01:58:06.750283 6615 server_base.cc:1034] running on GCE node
I20240502 01:58:06.750423 6615 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:58:06.750459 6615 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:58:06.750478 6615 hybrid_clock.cc:648] HybridClock initialized: now 1714615086750478 us; error 0 us; skew 500 ppm
I20240502 01:58:06.751411 6615 webserver.cc:460] Webserver started at http://127.6.117.253:45795/ using document root <none> and password file <none>
I20240502 01:58:06.751569 6615 fs_manager.cc:359] Metadata directory not provided
I20240502 01:58:06.751618 6615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:58:06.751682 6615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:58:06.752045 6615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714615076189642-6615-0/minicluster-data/master-1-root/instance:
uuid: "ccba1e41742b40948281c089a072e8f2"
format_stamp: "Formatted at 2024-05-02 01:58:06 on dist-test-slave-89cm"
I20240502 01:58:06.753691 6615 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240502 01:58:06.754503 7778 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.117.253" port: 39267 } has no permanent_uuid. Determining permanent_uuid...
I20240502 01:58:06.754799 7790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:06.755070 6615 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:06.755129 6615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714615076189642-6615-0/minicluster-data/master-1-root
uuid: "ccba1e41742b40948281c089a072e8f2"
format_stamp: "Formatted at 2024-05-02 01:58:06 on dist-test-slave-89cm"
I20240502 01:58:06.755190 6615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714615076189642-6615-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714615076189642-6615-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714615076189642-6615-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20240502 01:58:06.755681 7778 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.6.117.253:39267: Network error: Client connection negotiation failed: client connection to 127.6.117.253:39267: connect: Connection refused (error 111)
I20240502 01:58:06.760008 6615 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:58:06.760346 6615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:58:06.764292 6615 rpc_server.cc:307] RPC server started. Bound to: 127.6.117.253:39267
I20240502 01:58:06.764310 7843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.117.253:39267 every 8 connection(s)
I20240502 01:58:06.764956 6615 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 01:58:06.765137 7844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:06.766558 7844 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.117.254" port: 44289 } has no permanent_uuid. Determining permanent_uuid...
W20240502 01:58:06.766938 7846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:06.767218 7847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:06.767987 7849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:58:06.768365 6615 server_base.cc:1034] running on GCE node
I20240502 01:58:06.768523 6615 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:58:06.768563 6615 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:58:06.768582 6615 hybrid_clock.cc:648] HybridClock initialized: now 1714615086768582 us; error 0 us; skew 500 ppm
I20240502 01:58:06.769570 6615 webserver.cc:460] Webserver started at http://127.6.117.252:44981/ using document root <none> and password file <none>
I20240502 01:58:06.769748 6615 fs_manager.cc:359] Metadata directory not provided
I20240502 01:58:06.769819 6615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:58:06.769917 6615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:58:06.770442 6615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714615076189642-6615-0/minicluster-data/master-2-root/instance:
uuid: "583a10c367c944f798b885f9c08a6e49"
format_stamp: "Formatted at 2024-05-02 01:58:06 on dist-test-slave-89cm"
I20240502 01:58:06.772233 6615 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:06.772574 7844 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.117.253" port: 39267 } has no permanent_uuid. Determining permanent_uuid...
I20240502 01:58:06.773432 7856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:06.773700 6615 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:06.773773 6615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714615076189642-6615-0/minicluster-data/master-2-root
uuid: "583a10c367c944f798b885f9c08a6e49"
format_stamp: "Formatted at 2024-05-02 01:58:06 on dist-test-slave-89cm"
I20240502 01:58:06.773844 6615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714615076189642-6615-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714615076189642-6615-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714615076189642-6615-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:58:06.777076 7844 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.117.252" port: 44737 } has no permanent_uuid. Determining permanent_uuid...
W20240502 01:58:06.778184 7844 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.6.117.252:44737: Network error: Client connection negotiation failed: client connection to 127.6.117.252:44737: connect: Connection refused (error 111)
I20240502 01:58:06.780889 6615 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:58:06.781189 6615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:58:06.785001 6615 rpc_server.cc:307] RPC server started. Bound to: 127.6.117.252:44737
I20240502 01:58:06.785053 7909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.117.252:44737 every 8 connection(s)
I20240502 01:58:06.785418 6615 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 01:58:06.785863 7910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:06.786913 7910 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.117.254" port: 44289 } has no permanent_uuid. Determining permanent_uuid...
I20240502 01:58:06.791093 7910 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.117.253" port: 39267 } has no permanent_uuid. Determining permanent_uuid...
I20240502 01:58:06.794530 7910 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.117.252" port: 44737 } has no permanent_uuid. Determining permanent_uuid...
I20240502 01:58:06.799252 7910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 583a10c367c944f798b885f9c08a6e49: Bootstrap starting.
I20240502 01:58:06.800076 7910 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 583a10c367c944f798b885f9c08a6e49: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:06.801110 7910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 583a10c367c944f798b885f9c08a6e49: No bootstrap required, opened a new log
I20240502 01:58:06.801514 7910 raft_consensus.cc:348] T 00000000000000000000000000000000 P 583a10c367c944f798b885f9c08a6e49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1379406ff942b883b7fca5a80c5873" member_type: VOTER last_known_addr { host: "127.6.117.254" port: 44289 } } peers { permanent_uuid: "ccba1e41742b40948281c089a072e8f2" member_type: VOTER last_known_addr { host: "127.6.117.253" port: 39267 } } peers { permanent_uuid: "583a10c367c944f798b885f9c08a6e49" member_type: VOTER last_known_addr { host: "127.6.117.252" port: 44737 } }
I20240502 01:58:06.801632 7910 raft_consensus.cc:374] T 00000000000000000000000000000000 P 583a10c367c944f798b885f9c08a6e49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:06.801672 7910 raft_consensus.cc:729] T 00000000000000000000000000000000 P 583a10c367c944f798b885f9c08a6e49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 583a10c367c944f798b885f9c08a6e49, State: Initialized, Role: FOLLOWER
I20240502 01:58:06.801831 7910 consensus_queue.cc:260] T 00000000000000000000000000000000 P 583a10c367c944f798b885f9c08a6e49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1379406ff942b883b7fca5a80c5873" member_type: VOTER last_known_addr { host: "127.6.117.254" port: 44289 } } peers { permanent_uuid: "ccba1e41742b40948281c089a072e8f2" member_type: VOTER last_known_addr { host: "127.6.117.253" port: 39267 } } peers { permanent_uuid: "583a10c367c944f798b885f9c08a6e49" member_type: VOTER last_known_addr { host: "127.6.117.252" port: 44737 } }
I20240502 01:58:06.802163 7915 sys_catalog.cc:455] T 00000000000000000000000000000000 P 583a10c367c944f798b885f9c08a6e49 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1379406ff942b883b7fca5a80c5873" member_type: VOTER last_known_addr { host: "127.6.117.254" port: 44289 } } peers { permanent_uuid: "ccba1e41742b40948281c089a072e8f2" member_type: VOTER last_known_addr { host: "127.6.117.253" port: 39267 } } peers { permanent_uuid: "583a10c367c944f798b885f9c08a6e49" member_type: VOTER last_known_addr { host: "127.6.117.252" port: 44737 } } }
I20240502 01:58:06.802263 7915 sys_catalog.cc:458] T 00000000000000000000000000000000 P 583a10c367c944f798b885f9c08a6e49 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 01:58:06.802299 7910 sys_catalog.cc:564] T 00000000000000000000000000000000 P 583a10c367c944f798b885f9c08a6e49 [sys.catalog]: configured and running, proceeding with master startup.
W20240502 01:58:06.804378 7926 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 583a10c367c944f798b885f9c08a6e49: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240502 01:58:06.804446 7926 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240502 01:58:06.819849 7778 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.117.253" port: 39267 } attempt: 1
I20240502 01:58:06.824232 7778 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.117.252" port: 44737 } has no permanent_uuid. Determining permanent_uuid...
I20240502 01:58:06.828313 7844 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.117.252" port: 44737 } attempt: 1
I20240502 01:58:06.828727 7778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba1379406ff942b883b7fca5a80c5873: Bootstrap starting.
I20240502 01:58:06.829577 7778 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba1379406ff942b883b7fca5a80c5873: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:06.830792 7778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba1379406ff942b883b7fca5a80c5873: No bootstrap required, opened a new log
I20240502 01:58:06.831240 7778 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba1379406ff942b883b7fca5a80c5873 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1379406ff942b883b7fca5a80c5873" member_type: VOTER last_known_addr { host: "127.6.117.254" port: 44289 } } peers { permanent_uuid: "ccba1e41742b40948281c089a072e8f2" member_type: VOTER last_known_addr { host: "127.6.117.253" port: 39267 } } peers { permanent_uuid: "583a10c367c944f798b885f9c08a6e49" member_type: VOTER last_known_addr { host: "127.6.117.252" port: 44737 } }
I20240502 01:58:06.831365 7778 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba1379406ff942b883b7fca5a80c5873 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:06.831408 7778 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba1379406ff942b883b7fca5a80c5873 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba1379406ff942b883b7fca5a80c5873, State: Initialized, Role: FOLLOWER
I20240502 01:58:06.831593 7778 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba1379406ff942b883b7fca5a80c5873 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1379406ff942b883b7fca5a80c5873" member_type: VOTER last_known_addr { host: "127.6.117.254" port: 44289 } } peers { permanent_uuid: "ccba1e41742b40948281c089a072e8f2" member_type: VOTER last_known_addr { host: "127.6.117.253" port: 39267 } } peers { permanent_uuid: "583a10c367c944f798b885f9c08a6e49" member_type: VOTER last_known_addr { host: "127.6.117.252" port: 44737 } }
I20240502 01:58:06.831921 7928 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba1379406ff942b883b7fca5a80c5873 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1379406ff942b883b7fca5a80c5873" member_type: VOTER last_known_addr { host: "127.6.117.254" port: 44289 } } peers { permanent_uuid: "ccba1e41742b40948281c089a072e8f2" member_type: VOTER last_known_addr { host: "127.6.117.253" port: 39267 } } peers { permanent_uuid: "583a10c367c944f798b885f9c08a6e49" member_type: VOTER last_known_addr { host: "127.6.117.252" port: 44737 } } }
I20240502 01:58:06.832032 7928 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba1379406ff942b883b7fca5a80c5873 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 01:58:06.832080 7778 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba1379406ff942b883b7fca5a80c5873 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 01:58:06.832795 7844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccba1e41742b40948281c089a072e8f2: Bootstrap starting.
I20240502 01:58:06.833627 7844 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ccba1e41742b40948281c089a072e8f2: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:06.833623 6615 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
W20240502 01:58:06.834270 7939 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ba1379406ff942b883b7fca5a80c5873: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240502 01:58:06.834329 7939 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240502 01:58:06.834874 7844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccba1e41742b40948281c089a072e8f2: No bootstrap required, opened a new log
I20240502 01:58:06.835357 7844 raft_consensus.cc:348] T 00000000000000000000000000000000 P ccba1e41742b40948281c089a072e8f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1379406ff942b883b7fca5a80c5873" member_type: VOTER last_known_addr { host: "127.6.117.254" port: 44289 } } peers { permanent_uuid: "ccba1e41742b40948281c089a072e8f2" member_type: VOTER last_known_addr { host: "127.6.117.253" port: 39267 } } peers { permanent_uuid: "583a10c367c944f798b885f9c08a6e49" member_type: VOTER last_known_addr { host: "127.6.117.252" port: 44737 } }
I20240502 01:58:06.835475 7844 raft_consensus.cc:374] T 00000000000000000000000000000000 P ccba1e41742b40948281c089a072e8f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:06.835505 7844 raft_consensus.cc:729] T 00000000000000000000000000000000 P ccba1e41742b40948281c089a072e8f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccba1e41742b40948281c089a072e8f2, State: Initialized, Role: FOLLOWER
I20240502 01:58:06.835631 7844 consensus_queue.cc:260] T 00000000000000000000000000000000 P ccba1e41742b40948281c089a072e8f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1379406ff942b883b7fca5a80c5873" member_type: VOTER last_known_addr { host: "127.6.117.254" port: 44289 } } peers { permanent_uuid: "ccba1e41742b40948281c089a072e8f2" member_type: VOTER last_known_addr { host: "127.6.117.253" port: 39267 } } peers { permanent_uuid: "583a10c367c944f798b885f9c08a6e49" member_type: VOTER last_known_addr { host: "127.6.117.252" port: 44737 } }
I20240502 01:58:06.835922 7941 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccba1e41742b40948281c089a072e8f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1379406ff942b883b7fca5a80c5873" member_type: VOTER last_known_addr { host: "127.6.117.254" port: 44289 } } peers { permanent_uuid: "ccba1e41742b40948281c089a072e8f2" member_type: VOTER last_known_addr { host: "127.6.117.253" port: 39267 } } peers { permanent_uuid: "583a10c367c944f798b885f9c08a6e49" member_type: VOTER last_known_addr { host: "127.6.117.252" port: 44737 } } }
I20240502 01:58:06.836006 7941 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccba1e41742b40948281c089a072e8f2 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 01:58:06.836110 7844 sys_catalog.cc:564] T 00000000000000000000000000000000 P ccba1e41742b40948281c089a072e8f2 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 01:58:06.838068 6615 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
I20240502 01:58:06.838582 6615 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:58:06.838670 7952 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ccba1e41742b40948281c089a072e8f2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240502 01:58:06.838760 7952 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20240502 01:58:06.840819 7954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:06.841321 7953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:06.841642 7956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:58:06.842147 6615 server_base.cc:1034] running on GCE node
I20240502 01:58:06.842381 6615 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:58:06.842440 6615 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:58:06.842461 6615 hybrid_clock.cc:648] HybridClock initialized: now 1714615086842460 us; error 0 us; skew 500 ppm
I20240502 01:58:06.843410 6615 webserver.cc:460] Webserver started at http://127.6.117.193:35155/ using document root <none> and password file <none>
I20240502 01:58:06.843556 6615 fs_manager.cc:359] Metadata directory not provided
I20240502 01:58:06.843603 6615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:58:06.843677 6615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:58:06.844062 6615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714615076189642-6615-0/minicluster-data/ts-0-root/instance:
uuid: "a29ce75cc1f74a63a6c8f1d090e4127e"
format_stamp: "Formatted at 2024-05-02 01:58:06 on dist-test-slave-89cm"
I20240502 01:58:06.845468 6615 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240502 01:58:06.846493 7961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:06.846745 6615 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:06.846805 6615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714615076189642-6615-0/minicluster-data/ts-0-root
uuid: "a29ce75cc1f74a63a6c8f1d090e4127e"
format_stamp: "Formatted at 2024-05-02 01:58:06 on dist-test-slave-89cm"
I20240502 01:58:06.846853 6615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714615076189642-6615-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714615076189642-6615-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714615076189642-6615-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:58:06.858476 6615 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:58:06.858875 6615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:58:06.859161 6615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:58:06.859630 6615 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:58:06.859671 6615 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:06.859714 6615 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:58:06.859740 6615 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:06.864231 6615 rpc_server.cc:307] RPC server started. Bound to: 127.6.117.193:33401
I20240502 01:58:06.864296 8024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.117.193:33401 every 8 connection(s)
I20240502 01:58:06.865202 6615 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:58:06.868132 8035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:06.868324 8032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:06.868659 8033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:58:06.869082 6615 server_base.cc:1034] running on GCE node
I20240502 01:58:06.869503 8027 heartbeater.cc:347] Connected to a master server at 127.6.117.253:39267
I20240502 01:58:06.869601 8027 heartbeater.cc:464] Registering TS with master...
I20240502 01:58:06.869724 6615 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:58:06.869774 6615 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:58:06.869796 6615 hybrid_clock.cc:648] HybridClock initialized: now 1714615086869796 us; error 0 us; skew 500 ppm
I20240502 01:58:06.869817 8027 heartbeater.cc:511] Master 127.6.117.253:39267 requested a full tablet report, sending...
I20240502 01:58:06.870551 7808 ts_manager.cc:194] Registered new tserver with Master: a29ce75cc1f74a63a6c8f1d090e4127e (127.6.117.193:33401)
I20240502 01:58:06.870760 6615 webserver.cc:460] Webserver started at http://127.6.117.194:44125/ using document root <none> and password file <none>
I20240502 01:58:06.870952 6615 fs_manager.cc:359] Metadata directory not provided
I20240502 01:58:06.871004 6615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:58:06.871066 6615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:58:06.871206 8026 heartbeater.cc:347] Connected to a master server at 127.6.117.254:44289
I20240502 01:58:06.871268 8026 heartbeater.cc:464] Registering TS with master...
I20240502 01:58:06.871457 8026 heartbeater.cc:511] Master 127.6.117.254:44289 requested a full tablet report, sending...
I20240502 01:58:06.871493 6615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714615076189642-6615-0/minicluster-data/ts-1-root/instance:
uuid: "0314edf5ba0d486bafc4ace92679c4e3"
format_stamp: "Formatted at 2024-05-02 01:58:06 on dist-test-slave-89cm"
I20240502 01:58:06.872090 7742 ts_manager.cc:194] Registered new tserver with Master: a29ce75cc1f74a63a6c8f1d090e4127e (127.6.117.193:33401)
I20240502 01:58:06.873754 6615 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20240502 01:58:06.873785 8025 heartbeater.cc:347] Connected to a master server at 127.6.117.252:44737
I20240502 01:58:06.873855 8025 heartbeater.cc:464] Registering TS with master...
I20240502 01:58:06.874047 8025 heartbeater.cc:511] Master 127.6.117.252:44737 requested a full tablet report, sending...
I20240502 01:58:06.874650 7874 ts_manager.cc:194] Registered new tserver with Master: a29ce75cc1f74a63a6c8f1d090e4127e (127.6.117.193:33401)
I20240502 01:58:06.874794 8040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:06.875092 6615 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240502 01:58:06.875160 6615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714615076189642-6615-0/minicluster-data/ts-1-root
uuid: "0314edf5ba0d486bafc4ace92679c4e3"
format_stamp: "Formatted at 2024-05-02 01:58:06 on dist-test-slave-89cm"
I20240502 01:58:06.875211 6615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714615076189642-6615-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714615076189642-6615-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714615076189642-6615-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:58:06.895675 6615 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:58:06.896106 6615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:58:06.896416 6615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:58:06.896919 6615 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:58:06.896967 6615 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:06.897001 6615 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:58:06.897032 6615 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:06.901437 6615 rpc_server.cc:307] RPC server started. Bound to: 127.6.117.194:43329
I20240502 01:58:06.901521 8103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.117.194:43329 every 8 connection(s)
I20240502 01:58:06.902473 6615 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 01:58:06.903177 7915 raft_consensus.cc:482] T 00000000000000000000000000000000 P 583a10c367c944f798b885f9c08a6e49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:06.903293 7915 raft_consensus.cc:504] T 00000000000000000000000000000000 P 583a10c367c944f798b885f9c08a6e49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1379406ff942b883b7fca5a80c5873" member_type: VOTER last_known_addr { host: "127.6.117.254" port: 44289 } } peers { permanent_uuid: "ccba1e41742b40948281c089a072e8f2" member_type: VOTER last_known_addr { host: "127.6.117.253" port: 39267 } } peers { permanent_uuid: "583a10c367c944f798b885f9c08a6e49" member_type: VOTER last_known_addr { host: "127.6.117.252" port: 44737 } }
I20240502 01:58:06.903708 7915 leader_election.cc:288] T 00000000000000000000000000000000 P 583a10c367c944f798b885f9c08a6e49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba1379406ff942b883b7fca5a80c5873 (127.6.117.254:44289), ccba1e41742b40948281c089a072e8f2 (127.6.117.253:39267)
I20240502 01:58:06.904245 7752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "583a10c367c944f798b885f9c08a6e49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba1379406ff942b883b7fca5a80c5873" is_pre_election: true
I20240502 01:58:06.904397 7752 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ba1379406ff942b883b7fca5a80c5873 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583a10c367c944f798b885f9c08a6e49 in term 0.
I20240502 01:58:06.904842 7858 leader_election.cc:302] T 00000000000000000000000000000000 P 583a10c367c944f798b885f9c08a6e49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583a10c367c944f798b885f9c08a6e49, ba1379406ff942b883b7fca5a80c5873; no voters:
I20240502 01:58:06.905055 7818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "583a10c367c944f798b885f9c08a6e49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccba1e41742b40948281c089a072e8f2" is_pre_election: true
I20240502 01:58:06.905108 7915 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 583a10c367c944f798b885f9c08a6e49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:06.905179 7915 raft_consensus.cc:482] T 00000000000000000000000000000000 P 583a10c367c944f798b885f9c08a6e49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:06.905212 7915 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 583a10c367c944f798b885f9c08a6e49 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:06.905207 7818 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ccba1e41742b40948281c089a072e8f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583a10c367c944f798b885f9c08a6e49 in term 0.
I20240502 01:58:06.907840 8104 heartbeater.cc:347] Connected to a master server at 127.6.117.252:44737
I20240502 01:58:06.907958 8104 heartbeater.cc:464] Registering TS with master...
W20240502 01:58:06.908138 8111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:58:06.908174 8105 heartbeater.cc:347] Connected to a master server at 127.6.117.254:44289
I20240502 01:58:06.908241 8104 heartbeater.cc:511] Master 127.6.117.252:44737 requested a full tablet report, sending...
I20240502 01:58:06.908244 8105 heartbeater.cc:464] Registering TS with master...
I20240502 01:58:06.908252 8106 heartbeater.cc:347] Connected to a master server at 127.6.117.253:39267
I20240502 01:58:06.908432 8106 heartbeater.cc:464] Registering TS with master...
I20240502 01:58:06.908519 8105 heartbeater.cc:511] Master 127.6.117.254:44289 requested a full tablet report, sending...
W20240502 01:58:06.908548 8112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:58:06.908562 8106 heartbeater.cc:511] Master 127.6.117.253:39267 requested a full tablet report, sending...
W20240502 01:58:06.909000 8114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:58:06.909317 6615 server_base.cc:1034] running on GCE node
I20240502 01:58:06.909507 6615 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:58:06.909583 6615 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:58:06.909605 6615 hybrid_clock.cc:648] HybridClock initialized: now 1714615086909605 us; error 0 us; skew 500 ppm
I20240502 01:58:06.909667 7742 ts_manager.cc:194] Registered new tserver with Master: 0314edf5ba0d486bafc4ace92679c4e3 (127.6.117.194:43329)
I20240502 01:58:06.909672 7808 ts_manager.cc:194] Registered new tserver with Master: 0314edf5ba0d486bafc4ace92679c4e3 (127.6.117.194:43329)
I20240502 01:58:06.910620 6615 webserver.cc:460] Webserver started at http://127.6.117.195:35843/ using document root <none> and password file <none>
I20240502 01:58:06.910758 6615 fs_manager.cc:359] Metadata directory not provided
I20240502 01:58:06.910809 6615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:58:06.910882 6615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:58:06.911257 6615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714615076189642-6615-0/minicluster-data/ts-2-root/instance:
uuid: "6cc7355d5e4f4545a69798ecd7f9cb88"
format_stamp: "Formatted at 2024-05-02 01:58:06 on dist-test-slave-89cm"
I20240502 01:58:06.912714 6615 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240502 01:58:06.913527 8119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:06.913769 6615 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:06.913833 6615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714615076189642-6615-0/minicluster-data/ts-2-root
uuid: "6cc7355d5e4f4545a69798ecd7f9cb88"
format_stamp: "Formatted at 2024-05-02 01:58:06 on dist-test-slave-89cm"
I20240502 01:58:06.913893 6615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714615076189642-6615-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714615076189642-6615-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714615076189642-6615-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:58:06.920078 6615 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:58:06.920450 6615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:58:06.920713 6615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:58:06.921139 6615 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:58:06.921177 6615 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:06.921239 6615 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:58:06.921262 6615 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:06.925812 6615 rpc_server.cc:307] RPC server started. Bound to: 127.6.117.195:40641
I20240502 01:58:06.925833 8182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.117.195:40641 every 8 connection(s)
I20240502 01:58:06.930862 8183 heartbeater.cc:347] Connected to a master server at 127.6.117.252:44737
I20240502 01:58:06.930938 8185 heartbeater.cc:347] Connected to a master server at 127.6.117.253:39267
I20240502 01:58:06.930972 8183 heartbeater.cc:464] Registering TS with master...
I20240502 01:58:06.930989 8185 heartbeater.cc:464] Registering TS with master...
I20240502 01:58:06.931241 8184 heartbeater.cc:347] Connected to a master server at 127.6.117.254:44289
I20240502 01:58:06.931243 8185 heartbeater.cc:511] Master 127.6.117.253:39267 requested a full tablet report, sending...
I20240502 01:58:06.931244 8183 heartbeater.cc:511] Master 127.6.117.252:44737 requested a full tablet report, sending...
I20240502 01:58:06.931367 8184 heartbeater.cc:464] Registering TS with master...
I20240502 01:58:06.931579 8184 heartbeater.cc:511] Master 127.6.117.254:44289 requested a full tablet report, sending...
I20240502 01:58:06.932065 7808 ts_manager.cc:194] Registered new tserver with Master: 6cc7355d5e4f4545a69798ecd7f9cb88 (127.6.117.195:40641)
I20240502 01:58:06.932394 7742 ts_manager.cc:194] Registered new tserver with Master: 6cc7355d5e4f4545a69798ecd7f9cb88 (127.6.117.195:40641)
I20240502 01:58:06.970660 7915 raft_consensus.cc:504] T 00000000000000000000000000000000 P 583a10c367c944f798b885f9c08a6e49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1379406ff942b883b7fca5a80c5873" member_type: VOTER last_known_addr { host: "127.6.117.254" port: 44289 } } peers { permanent_uuid: "ccba1e41742b40948281c089a072e8f2" member_type: VOTER last_known_addr { host: "127.6.117.253" port: 39267 } } peers { permanent_uuid: "583a10c367c944f798b885f9c08a6e49" member_type: VOTER last_known_addr { host: "127.6.117.252" port: 44737 } }
I20240502 01:58:06.971266 7915 leader_election.cc:288] T 00000000000000000000000000000000 P 583a10c367c944f798b885f9c08a6e49 [CANDIDATE]: Term 1 election: Requested vote from peers ba1379406ff942b883b7fca5a80c5873 (127.6.117.254:44289), ccba1e41742b40948281c089a072e8f2 (127.6.117.253:39267)
I20240502 01:58:06.971570 7873 ts_manager.cc:194] Registered new tserver with Master: 6cc7355d5e4f4545a69798ecd7f9cb88 (127.6.117.195:40641)
I20240502 01:58:06.971766 7818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "583a10c367c944f798b885f9c08a6e49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccba1e41742b40948281c089a072e8f2"
I20240502 01:58:06.971782 7752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "583a10c367c944f798b885f9c08a6e49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba1379406ff942b883b7fca5a80c5873"
I20240502 01:58:06.971907 7818 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ccba1e41742b40948281c089a072e8f2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:06.971946 7752 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba1379406ff942b883b7fca5a80c5873 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:06.972370 7874 ts_manager.cc:194] Registered new tserver with Master: 0314edf5ba0d486bafc4ace92679c4e3 (127.6.117.194:43329)
I20240502 01:58:06.972558 6615 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.046264433s
I20240502 01:58:06.972973 7752 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ba1379406ff942b883b7fca5a80c5873 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 583a10c367c944f798b885f9c08a6e49 in term 1.
I20240502 01:58:06.973327 7858 leader_election.cc:302] T 00000000000000000000000000000000 P 583a10c367c944f798b885f9c08a6e49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583a10c367c944f798b885f9c08a6e49, ba1379406ff942b883b7fca5a80c5873; no voters:
I20240502 01:58:06.972973 7818 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ccba1e41742b40948281c089a072e8f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 583a10c367c944f798b885f9c08a6e49 in term 1.
I20240502 01:58:06.973850 7915 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 583a10c367c944f798b885f9c08a6e49 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:06.974061 7915 raft_consensus.cc:686] T 00000000000000000000000000000000 P 583a10c367c944f798b885f9c08a6e49 [term 1 LEADER]: Becoming Leader. State: Replica: 583a10c367c944f798b885f9c08a6e49, State: Running, Role: LEADER
I20240502 01:58:06.974212 7915 consensus_queue.cc:237] T 00000000000000000000000000000000 P 583a10c367c944f798b885f9c08a6e49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1379406ff942b883b7fca5a80c5873" member_type: VOTER last_known_addr { host: "127.6.117.254" port: 44289 } } peers { permanent_uuid: "ccba1e41742b40948281c089a072e8f2" member_type: VOTER last_known_addr { host: "127.6.117.253" port: 39267 } } peers { permanent_uuid: "583a10c367c944f798b885f9c08a6e49" member_type: VOTER last_known_addr { host: "127.6.117.252" port: 44737 } }
I20240502 01:58:06.975175 8191 sys_catalog.cc:455] T 00000000000000000000000000000000 P 583a10c367c944f798b885f9c08a6e49 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 583a10c367c944f798b885f9c08a6e49. Latest consensus state: current_term: 1 leader_uuid: "583a10c367c944f798b885f9c08a6e49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1379406ff942b883b7fca5a80c5873" member_type: VOTER last_known_addr { host: "127.6.117.254" port: 44289 } } peers { permanent_uuid: "ccba1e41742b40948281c089a072e8f2" member_type: VOTER last_known_addr { host: "127.6.117.253" port: 39267 } } peers { permanent_uuid: "583a10c367c944f798b885f9c08a6e49" member_type: VOTER last_known_addr { host: "127.6.117.252" port: 44737 } } }
I20240502 01:58:06.975277 8191 sys_catalog.cc:458] T 00000000000000000000000000000000 P 583a10c367c944f798b885f9c08a6e49 [sys.catalog]: This master's current role is: LEADER
I20240502 01:58:06.975548 8193 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 01:58:06.976431 8193 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 01:58:06.978194 7818 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ccba1e41742b40948281c089a072e8f2 [term 1 FOLLOWER]: Refusing update from remote peer 583a10c367c944f798b885f9c08a6e49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:06.978258 7752 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ba1379406ff942b883b7fca5a80c5873 [term 1 FOLLOWER]: Refusing update from remote peer 583a10c367c944f798b885f9c08a6e49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:06.978680 7915 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 583a10c367c944f798b885f9c08a6e49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccba1e41742b40948281c089a072e8f2" member_type: VOTER last_known_addr { host: "127.6.117.253" port: 39267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:06.978824 8191 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 583a10c367c944f798b885f9c08a6e49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba1379406ff942b883b7fca5a80c5873" member_type: VOTER last_known_addr { host: "127.6.117.254" port: 44289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:06.980141 7928 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba1379406ff942b883b7fca5a80c5873 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 583a10c367c944f798b885f9c08a6e49. Latest consensus state: current_term: 1 leader_uuid: "583a10c367c944f798b885f9c08a6e49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1379406ff942b883b7fca5a80c5873" member_type: VOTER last_known_addr { host: "127.6.117.254" port: 44289 } } peers { permanent_uuid: "ccba1e41742b40948281c089a072e8f2" member_type: VOTER last_known_addr { host: "127.6.117.253" port: 39267 } } peers { permanent_uuid: "583a10c367c944f798b885f9c08a6e49" member_type: VOTER last_known_addr { host: "127.6.117.252" port: 44737 } } }
I20240502 01:58:06.980245 7928 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba1379406ff942b883b7fca5a80c5873 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 01:58:06.980355 7941 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccba1e41742b40948281c089a072e8f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 583a10c367c944f798b885f9c08a6e49. Latest consensus state: current_term: 1 leader_uuid: "583a10c367c944f798b885f9c08a6e49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1379406ff942b883b7fca5a80c5873" member_type: VOTER last_known_addr { host: "127.6.117.254" port: 44289 } } peers { permanent_uuid: "ccba1e41742b40948281c089a072e8f2" member_type: VOTER last_known_addr { host: "127.6.117.253" port: 39267 } } peers { permanent_uuid: "583a10c367c944f798b885f9c08a6e49" member_type: VOTER last_known_addr { host: "127.6.117.252" port: 44737 } } }
I20240502 01:58:06.980470 7941 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccba1e41742b40948281c089a072e8f2 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 01:58:06.981609 8191 sys_catalog.cc:455] T 00000000000000000000000000000000 P 583a10c367c944f798b885f9c08a6e49 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "583a10c367c944f798b885f9c08a6e49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1379406ff942b883b7fca5a80c5873" member_type: VOTER last_known_addr { host: "127.6.117.254" port: 44289 } } peers { permanent_uuid: "ccba1e41742b40948281c089a072e8f2" member_type: VOTER last_known_addr { host: "127.6.117.253" port: 39267 } } peers { permanent_uuid: "583a10c367c944f798b885f9c08a6e49" member_type: VOTER last_known_addr { host: "127.6.117.252" port: 44737 } } }
I20240502 01:58:06.981695 8191 sys_catalog.cc:458] T 00000000000000000000000000000000 P 583a10c367c944f798b885f9c08a6e49 [sys.catalog]: This master's current role is: LEADER
I20240502 01:58:06.981796 8191 sys_catalog.cc:455] T 00000000000000000000000000000000 P 583a10c367c944f798b885f9c08a6e49 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "583a10c367c944f798b885f9c08a6e49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1379406ff942b883b7fca5a80c5873" member_type: VOTER last_known_addr { host: "127.6.117.254" port: 44289 } } peers { permanent_uuid: "ccba1e41742b40948281c089a072e8f2" member_type: VOTER last_known_addr { host: "127.6.117.253" port: 39267 } } peers { permanent_uuid: "583a10c367c944f798b885f9c08a6e49" member_type: VOTER last_known_addr { host: "127.6.117.252" port: 44737 } } }
I20240502 01:58:06.981849 8191 sys_catalog.cc:458] T 00000000000000000000000000000000 P 583a10c367c944f798b885f9c08a6e49 [sys.catalog]: This master's current role is: LEADER
I20240502 01:58:06.982048 7928 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba1379406ff942b883b7fca5a80c5873 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "583a10c367c944f798b885f9c08a6e49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1379406ff942b883b7fca5a80c5873" member_type: VOTER last_known_addr { host: "127.6.117.254" port: 44289 } } peers { permanent_uuid: "ccba1e41742b40948281c089a072e8f2" member_type: VOTER last_known_addr { host: "127.6.117.253" port: 39267 } } peers { permanent_uuid: "583a10c367c944f798b885f9c08a6e49" member_type: VOTER last_known_addr { host: "127.6.117.252" port: 44737 } } }
I20240502 01:58:06.982159 7928 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba1379406ff942b883b7fca5a80c5873 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 01:58:06.982615 8193 catalog_manager.cc:1347] Generated new cluster ID: bba66ff7248841938dffd02ad7a43cd9
I20240502 01:58:06.982610 7941 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccba1e41742b40948281c089a072e8f2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "583a10c367c944f798b885f9c08a6e49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1379406ff942b883b7fca5a80c5873" member_type: VOTER last_known_addr { host: "127.6.117.254" port: 44289 } } peers { permanent_uuid: "ccba1e41742b40948281c089a072e8f2" member_type: VOTER last_known_addr { host: "127.6.117.253" port: 39267 } } peers { permanent_uuid: "583a10c367c944f798b885f9c08a6e49" member_type: VOTER last_known_addr { host: "127.6.117.252" port: 44737 } } }
I20240502 01:58:06.982710 8193 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 01:58:06.982717 7941 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccba1e41742b40948281c089a072e8f2 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 01:58:07.000790 8193 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 01:58:07.001636 8193 catalog_manager.cc:1504] Loading token signing keys...
I20240502 01:58:07.009891 8193 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 583a10c367c944f798b885f9c08a6e49: Generated new TSK 0
I20240502 01:58:07.010170 8193 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 01:58:07.835151 7939 catalog_manager.cc:1259] Loaded cluster ID: bba66ff7248841938dffd02ad7a43cd9
I20240502 01:58:07.835238 7939 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ba1379406ff942b883b7fca5a80c5873: loading cluster ID for follower catalog manager: success
I20240502 01:58:07.836529 7939 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ba1379406ff942b883b7fca5a80c5873: acquiring CA information for follower catalog manager: success
I20240502 01:58:07.837100 7939 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ba1379406ff942b883b7fca5a80c5873: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240502 01:58:07.839303 7952 catalog_manager.cc:1259] Loaded cluster ID: bba66ff7248841938dffd02ad7a43cd9
I20240502 01:58:07.839360 7952 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ccba1e41742b40948281c089a072e8f2: loading cluster ID for follower catalog manager: success
I20240502 01:58:07.840956 7952 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ccba1e41742b40948281c089a072e8f2: acquiring CA information for follower catalog manager: success
I20240502 01:58:07.841549 7952 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ccba1e41742b40948281c089a072e8f2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240502 01:58:07.876320 7874 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58636
I20240502 01:58:07.973805 7874 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58660
I20240502 01:58:07.974655 7873 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58646
I20240502 01:58:08.804028 7925 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:58:08.877223 8025 heartbeater.cc:503] Master 127.6.117.252:44737 was elected leader, sending a full tablet report...
I20240502 01:58:08.974788 8183 heartbeater.cc:503] Master 127.6.117.252:44737 was elected leader, sending a full tablet report...
I20240502 01:58:08.975422 8104 heartbeater.cc:503] Master 127.6.117.252:44737 was elected leader, sending a full tablet report...
I20240502 01:58:09.131006 6615 test_util.cc:268] Using random seed: -279941639
I20240502 01:58:09.140319 7874 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60262:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\252\252\252*\004\001\000TUUU""\004\001\000\252\252\252*\004\001\000TUUU"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 01:58:09.153772 7989 tablet_service.cc:1467] Processing CreateTablet for tablet ff9b10bd4e4c468abf696ea0a992d748 (DEFAULT_TABLE table=test-workload [id=2778ee5336c44226ae814f4b6d61670b]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20240502 01:58:09.154193 7989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff9b10bd4e4c468abf696ea0a992d748. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:09.154320 8147 tablet_service.cc:1467] Processing CreateTablet for tablet f2570e6e4f294268a8ef862a8f22dc90 (DEFAULT_TABLE table=test-workload [id=2778ee5336c44226ae814f4b6d61670b]), partition=RANGE (key) PARTITION VALUES < 715827882
I20240502 01:58:09.154574 8147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2570e6e4f294268a8ef862a8f22dc90. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:09.154673 8068 tablet_service.cc:1467] Processing CreateTablet for tablet f3aab46168b743f690b431c176f9ee47 (DEFAULT_TABLE table=test-workload [id=2778ee5336c44226ae814f4b6d61670b]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20240502 01:58:09.154888 8068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3aab46168b743f690b431c176f9ee47. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:09.156977 8230 tablet_bootstrap.cc:492] T f2570e6e4f294268a8ef862a8f22dc90 P 6cc7355d5e4f4545a69798ecd7f9cb88: Bootstrap starting.
I20240502 01:58:09.157130 8232 tablet_bootstrap.cc:492] T f3aab46168b743f690b431c176f9ee47 P 0314edf5ba0d486bafc4ace92679c4e3: Bootstrap starting.
I20240502 01:58:09.157773 8230 tablet_bootstrap.cc:654] T f2570e6e4f294268a8ef862a8f22dc90 P 6cc7355d5e4f4545a69798ecd7f9cb88: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:09.157871 8232 tablet_bootstrap.cc:654] T f3aab46168b743f690b431c176f9ee47 P 0314edf5ba0d486bafc4ace92679c4e3: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:09.158375 8231 tablet_bootstrap.cc:492] T ff9b10bd4e4c468abf696ea0a992d748 P a29ce75cc1f74a63a6c8f1d090e4127e: Bootstrap starting.
I20240502 01:58:09.158715 8230 tablet_bootstrap.cc:492] T f2570e6e4f294268a8ef862a8f22dc90 P 6cc7355d5e4f4545a69798ecd7f9cb88: No bootstrap required, opened a new log
I20240502 01:58:09.158787 8230 ts_tablet_manager.cc:1367] T f2570e6e4f294268a8ef862a8f22dc90 P 6cc7355d5e4f4545a69798ecd7f9cb88: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:09.159137 8231 tablet_bootstrap.cc:654] T ff9b10bd4e4c468abf696ea0a992d748 P a29ce75cc1f74a63a6c8f1d090e4127e: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:09.159133 8230 raft_consensus.cc:348] T f2570e6e4f294268a8ef862a8f22dc90 P 6cc7355d5e4f4545a69798ecd7f9cb88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cc7355d5e4f4545a69798ecd7f9cb88" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 40641 } }
I20240502 01:58:09.159337 8230 raft_consensus.cc:374] T f2570e6e4f294268a8ef862a8f22dc90 P 6cc7355d5e4f4545a69798ecd7f9cb88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:09.159391 8230 raft_consensus.cc:729] T f2570e6e4f294268a8ef862a8f22dc90 P 6cc7355d5e4f4545a69798ecd7f9cb88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc7355d5e4f4545a69798ecd7f9cb88, State: Initialized, Role: FOLLOWER
I20240502 01:58:09.159519 8230 consensus_queue.cc:260] T f2570e6e4f294268a8ef862a8f22dc90 P 6cc7355d5e4f4545a69798ecd7f9cb88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cc7355d5e4f4545a69798ecd7f9cb88" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 40641 } }
I20240502 01:58:09.159603 8230 raft_consensus.cc:388] T f2570e6e4f294268a8ef862a8f22dc90 P 6cc7355d5e4f4545a69798ecd7f9cb88 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:58:09.159626 8230 raft_consensus.cc:482] T f2570e6e4f294268a8ef862a8f22dc90 P 6cc7355d5e4f4545a69798ecd7f9cb88 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:58:09.159669 8230 raft_consensus.cc:3037] T f2570e6e4f294268a8ef862a8f22dc90 P 6cc7355d5e4f4545a69798ecd7f9cb88 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:09.160028 8232 tablet_bootstrap.cc:492] T f3aab46168b743f690b431c176f9ee47 P 0314edf5ba0d486bafc4ace92679c4e3: No bootstrap required, opened a new log
I20240502 01:58:09.160110 8232 ts_tablet_manager.cc:1367] T f3aab46168b743f690b431c176f9ee47 P 0314edf5ba0d486bafc4ace92679c4e3: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:58:09.160465 8230 raft_consensus.cc:504] T f2570e6e4f294268a8ef862a8f22dc90 P 6cc7355d5e4f4545a69798ecd7f9cb88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cc7355d5e4f4545a69798ecd7f9cb88" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 40641 } }
I20240502 01:58:09.160557 8232 raft_consensus.cc:348] T f3aab46168b743f690b431c176f9ee47 P 0314edf5ba0d486bafc4ace92679c4e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0314edf5ba0d486bafc4ace92679c4e3" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 43329 } }
I20240502 01:58:09.160656 8230 leader_election.cc:302] T f2570e6e4f294268a8ef862a8f22dc90 P 6cc7355d5e4f4545a69798ecd7f9cb88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6cc7355d5e4f4545a69798ecd7f9cb88; no voters:
I20240502 01:58:09.160691 8232 raft_consensus.cc:374] T f3aab46168b743f690b431c176f9ee47 P 0314edf5ba0d486bafc4ace92679c4e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:09.160727 8232 raft_consensus.cc:729] T f3aab46168b743f690b431c176f9ee47 P 0314edf5ba0d486bafc4ace92679c4e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0314edf5ba0d486bafc4ace92679c4e3, State: Initialized, Role: FOLLOWER
I20240502 01:58:09.160616 8231 tablet_bootstrap.cc:492] T ff9b10bd4e4c468abf696ea0a992d748 P a29ce75cc1f74a63a6c8f1d090e4127e: No bootstrap required, opened a new log
I20240502 01:58:09.160846 8230 leader_election.cc:288] T f2570e6e4f294268a8ef862a8f22dc90 P 6cc7355d5e4f4545a69798ecd7f9cb88 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:58:09.160885 8231 ts_tablet_manager.cc:1367] T ff9b10bd4e4c468abf696ea0a992d748 P a29ce75cc1f74a63a6c8f1d090e4127e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:58:09.160919 8232 consensus_queue.cc:260] T f3aab46168b743f690b431c176f9ee47 P 0314edf5ba0d486bafc4ace92679c4e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0314edf5ba0d486bafc4ace92679c4e3" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 43329 } }
I20240502 01:58:09.160997 8232 raft_consensus.cc:388] T f3aab46168b743f690b431c176f9ee47 P 0314edf5ba0d486bafc4ace92679c4e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:58:09.161028 8232 raft_consensus.cc:482] T f3aab46168b743f690b431c176f9ee47 P 0314edf5ba0d486bafc4ace92679c4e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:58:09.161108 8232 raft_consensus.cc:3037] T f3aab46168b743f690b431c176f9ee47 P 0314edf5ba0d486bafc4ace92679c4e3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:09.161211 8230 ts_tablet_manager.cc:1398] T f2570e6e4f294268a8ef862a8f22dc90 P 6cc7355d5e4f4545a69798ecd7f9cb88: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240502 01:58:09.161342 8231 raft_consensus.cc:348] T ff9b10bd4e4c468abf696ea0a992d748 P a29ce75cc1f74a63a6c8f1d090e4127e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a29ce75cc1f74a63a6c8f1d090e4127e" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 33401 } }
I20240502 01:58:09.161479 8231 raft_consensus.cc:374] T ff9b10bd4e4c468abf696ea0a992d748 P a29ce75cc1f74a63a6c8f1d090e4127e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:09.161548 8231 raft_consensus.cc:729] T ff9b10bd4e4c468abf696ea0a992d748 P a29ce75cc1f74a63a6c8f1d090e4127e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29ce75cc1f74a63a6c8f1d090e4127e, State: Initialized, Role: FOLLOWER
I20240502 01:58:09.161736 8231 consensus_queue.cc:260] T ff9b10bd4e4c468abf696ea0a992d748 P a29ce75cc1f74a63a6c8f1d090e4127e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a29ce75cc1f74a63a6c8f1d090e4127e" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 33401 } }
I20240502 01:58:09.161875 8231 raft_consensus.cc:388] T ff9b10bd4e4c468abf696ea0a992d748 P a29ce75cc1f74a63a6c8f1d090e4127e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:58:09.161952 8231 raft_consensus.cc:482] T ff9b10bd4e4c468abf696ea0a992d748 P a29ce75cc1f74a63a6c8f1d090e4127e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:58:09.162009 8232 raft_consensus.cc:504] T f3aab46168b743f690b431c176f9ee47 P 0314edf5ba0d486bafc4ace92679c4e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0314edf5ba0d486bafc4ace92679c4e3" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 43329 } }
I20240502 01:58:09.162148 8232 leader_election.cc:302] T f3aab46168b743f690b431c176f9ee47 P 0314edf5ba0d486bafc4ace92679c4e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0314edf5ba0d486bafc4ace92679c4e3; no voters:
I20240502 01:58:09.162322 8232 leader_election.cc:288] T f3aab46168b743f690b431c176f9ee47 P 0314edf5ba0d486bafc4ace92679c4e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:58:09.162405 8238 raft_consensus.cc:2781] T f3aab46168b743f690b431c176f9ee47 P 0314edf5ba0d486bafc4ace92679c4e3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:09.162556 8238 raft_consensus.cc:686] T f3aab46168b743f690b431c176f9ee47 P 0314edf5ba0d486bafc4ace92679c4e3 [term 1 LEADER]: Becoming Leader. State: Replica: 0314edf5ba0d486bafc4ace92679c4e3, State: Running, Role: LEADER
I20240502 01:58:09.162681 8232 ts_tablet_manager.cc:1398] T f3aab46168b743f690b431c176f9ee47 P 0314edf5ba0d486bafc4ace92679c4e3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:58:09.162726 8238 consensus_queue.cc:237] T f3aab46168b743f690b431c176f9ee47 P 0314edf5ba0d486bafc4ace92679c4e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0314edf5ba0d486bafc4ace92679c4e3" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 43329 } }
I20240502 01:58:09.162036 8231 raft_consensus.cc:3037] T ff9b10bd4e4c468abf696ea0a992d748 P a29ce75cc1f74a63a6c8f1d090e4127e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:09.162894 8237 raft_consensus.cc:2781] T f2570e6e4f294268a8ef862a8f22dc90 P 6cc7355d5e4f4545a69798ecd7f9cb88 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:09.163340 8237 raft_consensus.cc:686] T f2570e6e4f294268a8ef862a8f22dc90 P 6cc7355d5e4f4545a69798ecd7f9cb88 [term 1 LEADER]: Becoming Leader. State: Replica: 6cc7355d5e4f4545a69798ecd7f9cb88, State: Running, Role: LEADER
I20240502 01:58:09.163516 8237 consensus_queue.cc:237] T f2570e6e4f294268a8ef862a8f22dc90 P 6cc7355d5e4f4545a69798ecd7f9cb88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cc7355d5e4f4545a69798ecd7f9cb88" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 40641 } }
I20240502 01:58:09.163949 8231 raft_consensus.cc:504] T ff9b10bd4e4c468abf696ea0a992d748 P a29ce75cc1f74a63a6c8f1d090e4127e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a29ce75cc1f74a63a6c8f1d090e4127e" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 33401 } }
I20240502 01:58:09.164026 7874 catalog_manager.cc:5484] T f3aab46168b743f690b431c176f9ee47 P 0314edf5ba0d486bafc4ace92679c4e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0314edf5ba0d486bafc4ace92679c4e3 (127.6.117.194). New cstate: current_term: 1 leader_uuid: "0314edf5ba0d486bafc4ace92679c4e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0314edf5ba0d486bafc4ace92679c4e3" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 43329 } health_report { overall_health: HEALTHY } } }
I20240502 01:58:09.164237 8231 leader_election.cc:302] T ff9b10bd4e4c468abf696ea0a992d748 P a29ce75cc1f74a63a6c8f1d090e4127e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a29ce75cc1f74a63a6c8f1d090e4127e; no voters:
I20240502 01:58:09.164499 8231 leader_election.cc:288] T ff9b10bd4e4c468abf696ea0a992d748 P a29ce75cc1f74a63a6c8f1d090e4127e [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:58:09.164719 7873 catalog_manager.cc:5484] T f2570e6e4f294268a8ef862a8f22dc90 P 6cc7355d5e4f4545a69798ecd7f9cb88 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6cc7355d5e4f4545a69798ecd7f9cb88 (127.6.117.195). New cstate: current_term: 1 leader_uuid: "6cc7355d5e4f4545a69798ecd7f9cb88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cc7355d5e4f4545a69798ecd7f9cb88" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 40641 } health_report { overall_health: HEALTHY } } }
I20240502 01:58:09.164812 8231 ts_tablet_manager.cc:1398] T ff9b10bd4e4c468abf696ea0a992d748 P a29ce75cc1f74a63a6c8f1d090e4127e: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:58:09.167989 8244 raft_consensus.cc:2781] T ff9b10bd4e4c468abf696ea0a992d748 P a29ce75cc1f74a63a6c8f1d090e4127e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:09.168715 8244 raft_consensus.cc:686] T ff9b10bd4e4c468abf696ea0a992d748 P a29ce75cc1f74a63a6c8f1d090e4127e [term 1 LEADER]: Becoming Leader. State: Replica: a29ce75cc1f74a63a6c8f1d090e4127e, State: Running, Role: LEADER
I20240502 01:58:09.168920 8244 consensus_queue.cc:237] T ff9b10bd4e4c468abf696ea0a992d748 P a29ce75cc1f74a63a6c8f1d090e4127e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a29ce75cc1f74a63a6c8f1d090e4127e" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 33401 } }
I20240502 01:58:09.170332 7873 catalog_manager.cc:5484] T ff9b10bd4e4c468abf696ea0a992d748 P a29ce75cc1f74a63a6c8f1d090e4127e reported cstate change: term changed from 0 to 1, leader changed from <none> to a29ce75cc1f74a63a6c8f1d090e4127e (127.6.117.193). New cstate: current_term: 1 leader_uuid: "a29ce75cc1f74a63a6c8f1d090e4127e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a29ce75cc1f74a63a6c8f1d090e4127e" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 33401 } health_report { overall_health: HEALTHY } } }
I20240502 01:58:09.179093 6615 master.cc:537] Master@127.6.117.252:44737 shutting down...
I20240502 01:58:09.182097 6615 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 583a10c367c944f798b885f9c08a6e49 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:58:09.182312 6615 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 583a10c367c944f798b885f9c08a6e49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:09.182365 6615 tablet_replica.cc:330] T 00000000000000000000000000000000 P 583a10c367c944f798b885f9c08a6e49: stopping tablet replica
I20240502 01:58:09.196141 6615 master.cc:552] Master@127.6.117.252:44737 shutdown complete.
W20240502 01:58:10.172725 8104 heartbeater.cc:644] Failed to heartbeat to 127.6.117.252:44737 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.6.117.252:44737: connect: Connection refused (error 111)
I20240502 01:58:10.675581 8254 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba1379406ff942b883b7fca5a80c5873 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 583a10c367c944f798b885f9c08a6e49)
I20240502 01:58:10.675657 8255 raft_consensus.cc:482] T 00000000000000000000000000000000 P ccba1e41742b40948281c089a072e8f2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 583a10c367c944f798b885f9c08a6e49)
I20240502 01:58:10.675766 8255 raft_consensus.cc:504] T 00000000000000000000000000000000 P ccba1e41742b40948281c089a072e8f2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1379406ff942b883b7fca5a80c5873" member_type: VOTER last_known_addr { host: "127.6.117.254" port: 44289 } } peers { permanent_uuid: "ccba1e41742b40948281c089a072e8f2" member_type: VOTER last_known_addr { host: "127.6.117.253" port: 39267 } } peers { permanent_uuid: "583a10c367c944f798b885f9c08a6e49" member_type: VOTER last_known_addr { host: "127.6.117.252" port: 44737 } }
I20240502 01:58:10.675768 8254 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba1379406ff942b883b7fca5a80c5873 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1379406ff942b883b7fca5a80c5873" member_type: VOTER last_known_addr { host: "127.6.117.254" port: 44289 } } peers { permanent_uuid: "ccba1e41742b40948281c089a072e8f2" member_type: VOTER last_known_addr { host: "127.6.117.253" port: 39267 } } peers { permanent_uuid: "583a10c367c944f798b885f9c08a6e49" member_type: VOTER last_known_addr { host: "127.6.117.252" port: 44737 } }
I20240502 01:58:10.676213 8254 leader_election.cc:288] T 00000000000000000000000000000000 P ba1379406ff942b883b7fca5a80c5873 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ccba1e41742b40948281c089a072e8f2 (127.6.117.253:39267), 583a10c367c944f798b885f9c08a6e49 (127.6.117.252:44737)
I20240502 01:58:10.676220 8255 leader_election.cc:288] T 00000000000000000000000000000000 P ccba1e41742b40948281c089a072e8f2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ba1379406ff942b883b7fca5a80c5873 (127.6.117.254:44289), 583a10c367c944f798b885f9c08a6e49 (127.6.117.252:44737)
I20240502 01:58:10.676699 7818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ba1379406ff942b883b7fca5a80c5873" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "ccba1e41742b40948281c089a072e8f2" is_pre_election: true
I20240502 01:58:10.676816 7818 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ccba1e41742b40948281c089a072e8f2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba1379406ff942b883b7fca5a80c5873 in term 1.
I20240502 01:58:10.676826 7752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ccba1e41742b40948281c089a072e8f2" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "ba1379406ff942b883b7fca5a80c5873" is_pre_election: true
I20240502 01:58:10.677141 7752 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ba1379406ff942b883b7fca5a80c5873 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccba1e41742b40948281c089a072e8f2 in term 1.
W20240502 01:58:10.677507 7793 leader_election.cc:334] T 00000000000000000000000000000000 P ccba1e41742b40948281c089a072e8f2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 583a10c367c944f798b885f9c08a6e49 (127.6.117.252:44737): Network error: Client connection negotiation failed: client connection to 127.6.117.252:44737: connect: Connection refused (error 111)
I20240502 01:58:10.677568 7727 leader_election.cc:302] T 00000000000000000000000000000000 P ba1379406ff942b883b7fca5a80c5873 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba1379406ff942b883b7fca5a80c5873, ccba1e41742b40948281c089a072e8f2; no voters:
I20240502 01:58:10.677646 7793 leader_election.cc:302] T 00000000000000000000000000000000 P ccba1e41742b40948281c089a072e8f2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ba1379406ff942b883b7fca5a80c5873, ccba1e41742b40948281c089a072e8f2; no voters: 583a10c367c944f798b885f9c08a6e49
W20240502 01:58:10.677711 7727 leader_election.cc:334] T 00000000000000000000000000000000 P ba1379406ff942b883b7fca5a80c5873 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 583a10c367c944f798b885f9c08a6e49 (127.6.117.252:44737): Network error: Client connection negotiation failed: client connection to 127.6.117.252:44737: connect: Connection refused (error 111)
I20240502 01:58:10.677717 8254 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba1379406ff942b883b7fca5a80c5873 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 01:58:10.677783 8254 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba1379406ff942b883b7fca5a80c5873 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 583a10c367c944f798b885f9c08a6e49)
I20240502 01:58:10.677829 8254 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba1379406ff942b883b7fca5a80c5873 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:10.677850 8255 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ccba1e41742b40948281c089a072e8f2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 01:58:10.677899 8255 raft_consensus.cc:482] T 00000000000000000000000000000000 P ccba1e41742b40948281c089a072e8f2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 583a10c367c944f798b885f9c08a6e49)
I20240502 01:58:10.677940 8255 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ccba1e41742b40948281c089a072e8f2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:10.678903 8254 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba1379406ff942b883b7fca5a80c5873 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1379406ff942b883b7fca5a80c5873" member_type: VOTER last_known_addr { host: "127.6.117.254" port: 44289 } } peers { permanent_uuid: "ccba1e41742b40948281c089a072e8f2" member_type: VOTER last_known_addr { host: "127.6.117.253" port: 39267 } } peers { permanent_uuid: "583a10c367c944f798b885f9c08a6e49" member_type: VOTER last_known_addr { host: "127.6.117.252" port: 44737 } }
I20240502 01:58:10.678903 8255 raft_consensus.cc:504] T 00000000000000000000000000000000 P ccba1e41742b40948281c089a072e8f2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1379406ff942b883b7fca5a80c5873" member_type: VOTER last_known_addr { host: "127.6.117.254" port: 44289 } } peers { permanent_uuid: "ccba1e41742b40948281c089a072e8f2" member_type: VOTER last_known_addr { host: "127.6.117.253" port: 39267 } } peers { permanent_uuid: "583a10c367c944f798b885f9c08a6e49" member_type: VOTER last_known_addr { host: "127.6.117.252" port: 44737 } }
I20240502 01:58:10.679280 8255 leader_election.cc:288] T 00000000000000000000000000000000 P ccba1e41742b40948281c089a072e8f2 [CANDIDATE]: Term 2 election: Requested vote from peers ba1379406ff942b883b7fca5a80c5873 (127.6.117.254:44289), 583a10c367c944f798b885f9c08a6e49 (127.6.117.252:44737)
I20240502 01:58:10.679289 8254 leader_election.cc:288] T 00000000000000000000000000000000 P ba1379406ff942b883b7fca5a80c5873 [CANDIDATE]: Term 2 election: Requested vote from peers ccba1e41742b40948281c089a072e8f2 (127.6.117.253:39267), 583a10c367c944f798b885f9c08a6e49 (127.6.117.252:44737)
I20240502 01:58:10.679809 7818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ba1379406ff942b883b7fca5a80c5873" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "ccba1e41742b40948281c089a072e8f2"
I20240502 01:58:10.679939 7818 raft_consensus.cc:2379] T 00000000000000000000000000000000 P ccba1e41742b40948281c089a072e8f2 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate ba1379406ff942b883b7fca5a80c5873 in current term 2: Already voted for candidate ccba1e41742b40948281c089a072e8f2 in this term.
I20240502 01:58:10.679973 7752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ccba1e41742b40948281c089a072e8f2" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "ba1379406ff942b883b7fca5a80c5873"
I20240502 01:58:10.680089 7752 raft_consensus.cc:2379] T 00000000000000000000000000000000 P ba1379406ff942b883b7fca5a80c5873 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate ccba1e41742b40948281c089a072e8f2 in current term 2: Already voted for candidate ba1379406ff942b883b7fca5a80c5873 in this term.
W20240502 01:58:10.680294 7793 leader_election.cc:334] T 00000000000000000000000000000000 P ccba1e41742b40948281c089a072e8f2 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 583a10c367c944f798b885f9c08a6e49 (127.6.117.252:44737): Network error: Client connection negotiation failed: client connection to 127.6.117.252:44737: connect: Connection refused (error 111)
I20240502 01:58:10.680444 7793 leader_election.cc:302] T 00000000000000000000000000000000 P ccba1e41742b40948281c089a072e8f2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ccba1e41742b40948281c089a072e8f2; no voters: 583a10c367c944f798b885f9c08a6e49, ba1379406ff942b883b7fca5a80c5873
W20240502 01:58:10.680480 7727 leader_election.cc:334] T 00000000000000000000000000000000 P ba1379406ff942b883b7fca5a80c5873 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 583a10c367c944f798b885f9c08a6e49 (127.6.117.252:44737): Network error: Client connection negotiation failed: client connection to 127.6.117.252:44737: connect: Connection refused (error 111)
I20240502 01:58:10.680532 7727 leader_election.cc:302] T 00000000000000000000000000000000 P ba1379406ff942b883b7fca5a80c5873 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba1379406ff942b883b7fca5a80c5873; no voters: 583a10c367c944f798b885f9c08a6e49, ccba1e41742b40948281c089a072e8f2
I20240502 01:58:10.680586 8255 raft_consensus.cc:2726] T 00000000000000000000000000000000 P ccba1e41742b40948281c089a072e8f2 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240502 01:58:10.680629 8254 raft_consensus.cc:2726] T 00000000000000000000000000000000 P ba1379406ff942b883b7fca5a80c5873 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240502 01:58:12.279284 8266 raft_consensus.cc:482] T 00000000000000000000000000000000 P ccba1e41742b40948281c089a072e8f2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:12.279439 8266 raft_consensus.cc:504] T 00000000000000000000000000000000 P ccba1e41742b40948281c089a072e8f2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1379406ff942b883b7fca5a80c5873" member_type: VOTER last_known_addr { host: "127.6.117.254" port: 44289 } } peers { permanent_uuid: "ccba1e41742b40948281c089a072e8f2" member_type: VOTER last_known_addr { host: "127.6.117.253" port: 39267 } } peers { permanent_uuid: "583a10c367c944f798b885f9c08a6e49" member_type: VOTER last_known_addr { host: "127.6.117.252" port: 44737 } }
I20240502 01:58:12.279844 8266 leader_election.cc:288] T 00000000000000000000000000000000 P ccba1e41742b40948281c089a072e8f2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ba1379406ff942b883b7fca5a80c5873 (127.6.117.254:44289), 583a10c367c944f798b885f9c08a6e49 (127.6.117.252:44737)
I20240502 01:58:12.280339 7752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ccba1e41742b40948281c089a072e8f2" candidate_term: 3 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "ba1379406ff942b883b7fca5a80c5873" is_pre_election: true
I20240502 01:58:12.280501 7752 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ba1379406ff942b883b7fca5a80c5873 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccba1e41742b40948281c089a072e8f2 in term 2.
W20240502 01:58:12.280634 7793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.117.252:44737: connect: Connection refused (error 111) [suppressed 11 similar messages]
I20240502 01:58:12.280977 7793 leader_election.cc:302] T 00000000000000000000000000000000 P ccba1e41742b40948281c089a072e8f2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba1379406ff942b883b7fca5a80c5873, ccba1e41742b40948281c089a072e8f2; no voters:
I20240502 01:58:12.281132 8266 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ccba1e41742b40948281c089a072e8f2 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 01:58:12.281194 8266 raft_consensus.cc:482] T 00000000000000000000000000000000 P ccba1e41742b40948281c089a072e8f2 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:12.281246 8266 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ccba1e41742b40948281c089a072e8f2 [term 2 FOLLOWER]: Advancing to term 3
W20240502 01:58:12.281492 7793 leader_election.cc:334] T 00000000000000000000000000000000 P ccba1e41742b40948281c089a072e8f2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 583a10c367c944f798b885f9c08a6e49 (127.6.117.252:44737): Network error: Client connection negotiation failed: client connection to 127.6.117.252:44737: connect: Connection refused (error 111)
I20240502 01:58:12.282233 8266 raft_consensus.cc:504] T 00000000000000000000000000000000 P ccba1e41742b40948281c089a072e8f2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1379406ff942b883b7fca5a80c5873" member_type: VOTER last_known_addr { host: "127.6.117.254" port: 44289 } } peers { permanent_uuid: "ccba1e41742b40948281c089a072e8f2" member_type: VOTER last_known_addr { host: "127.6.117.253" port: 39267 } } peers { permanent_uuid: "583a10c367c944f798b885f9c08a6e49" member_type: VOTER last_known_addr { host: "127.6.117.252" port: 44737 } }
I20240502 01:58:12.282541 8266 leader_election.cc:288] T 00000000000000000000000000000000 P ccba1e41742b40948281c089a072e8f2 [CANDIDATE]: Term 3 election: Requested vote from peers ba1379406ff942b883b7fca5a80c5873 (127.6.117.254:44289), 583a10c367c944f798b885f9c08a6e49 (127.6.117.252:44737)
I20240502 01:58:12.282841 7752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ccba1e41742b40948281c089a072e8f2" candidate_term: 3 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "ba1379406ff942b883b7fca5a80c5873"
I20240502 01:58:12.282948 7752 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba1379406ff942b883b7fca5a80c5873 [term 2 FOLLOWER]: Advancing to term 3
W20240502 01:58:12.283489 7793 leader_election.cc:334] T 00000000000000000000000000000000 P ccba1e41742b40948281c089a072e8f2 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 583a10c367c944f798b885f9c08a6e49 (127.6.117.252:44737): Network error: Client connection negotiation failed: client connection to 127.6.117.252:44737: connect: Connection refused (error 111)
I20240502 01:58:12.283705 7752 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ba1379406ff942b883b7fca5a80c5873 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccba1e41742b40948281c089a072e8f2 in term 3.
I20240502 01:58:12.283953 7793 leader_election.cc:302] T 00000000000000000000000000000000 P ccba1e41742b40948281c089a072e8f2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ba1379406ff942b883b7fca5a80c5873, ccba1e41742b40948281c089a072e8f2; no voters: 583a10c367c944f798b885f9c08a6e49
I20240502 01:58:12.284103 8266 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ccba1e41742b40948281c089a072e8f2 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:58:12.284318 8266 raft_consensus.cc:686] T 00000000000000000000000000000000 P ccba1e41742b40948281c089a072e8f2 [term 3 LEADER]: Becoming Leader. State: Replica: ccba1e41742b40948281c089a072e8f2, State: Running, Role: LEADER
I20240502 01:58:12.284466 8266 consensus_queue.cc:237] T 00000000000000000000000000000000 P ccba1e41742b40948281c089a072e8f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1379406ff942b883b7fca5a80c5873" member_type: VOTER last_known_addr { host: "127.6.117.254" port: 44289 } } peers { permanent_uuid: "ccba1e41742b40948281c089a072e8f2" member_type: VOTER last_known_addr { host: "127.6.117.253" port: 39267 } } peers { permanent_uuid: "583a10c367c944f798b885f9c08a6e49" member_type: VOTER last_known_addr { host: "127.6.117.252" port: 44737 } }
I20240502 01:58:12.285092 8269 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccba1e41742b40948281c089a072e8f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ccba1e41742b40948281c089a072e8f2. Latest consensus state: current_term: 3 leader_uuid: "ccba1e41742b40948281c089a072e8f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1379406ff942b883b7fca5a80c5873" member_type: VOTER last_known_addr { host: "127.6.117.254" port: 44289 } } peers { permanent_uuid: "ccba1e41742b40948281c089a072e8f2" member_type: VOTER last_known_addr { host: "127.6.117.253" port: 39267 } } peers { permanent_uuid: "583a10c367c944f798b885f9c08a6e49" member_type: VOTER last_known_addr { host: "127.6.117.252" port: 44737 } } }
I20240502 01:58:12.285216 8269 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccba1e41742b40948281c089a072e8f2 [sys.catalog]: This master's current role is: LEADER
I20240502 01:58:12.285497 8271 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 01:58:12.286597 8271 catalog_manager.cc:669] Loaded metadata for table test-workload [id=2778ee5336c44226ae814f4b6d61670b]
I20240502 01:58:12.287305 8271 tablet_loader.cc:96] loaded metadata for tablet f2570e6e4f294268a8ef862a8f22dc90 (table test-workload [id=2778ee5336c44226ae814f4b6d61670b])
I20240502 01:58:12.287496 8271 tablet_loader.cc:96] loaded metadata for tablet f3aab46168b743f690b431c176f9ee47 (table test-workload [id=2778ee5336c44226ae814f4b6d61670b])
I20240502 01:58:12.287662 8271 tablet_loader.cc:96] loaded metadata for tablet ff9b10bd4e4c468abf696ea0a992d748 (table test-workload [id=2778ee5336c44226ae814f4b6d61670b])
I20240502 01:58:12.287755 8271 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 01:58:12.288141 8271 catalog_manager.cc:1259] Loaded cluster ID: bba66ff7248841938dffd02ad7a43cd9
I20240502 01:58:12.288184 8271 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 01:58:12.288738 8271 catalog_manager.cc:1504] Loading token signing keys...
I20240502 01:58:12.289135 8271 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ccba1e41742b40948281c089a072e8f2: Loaded TSK: 0
I20240502 01:58:12.289615 8271 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 01:58:12.716333 7752 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ba1379406ff942b883b7fca5a80c5873 [term 3 FOLLOWER]: Refusing update from remote peer ccba1e41742b40948281c089a072e8f2: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 3 index: 10. (index mismatch)
I20240502 01:58:12.716799 8269 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ccba1e41742b40948281c089a072e8f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba1379406ff942b883b7fca5a80c5873" member_type: VOTER last_known_addr { host: "127.6.117.254" port: 44289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:58:12.717859 8272 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba1379406ff942b883b7fca5a80c5873 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ccba1e41742b40948281c089a072e8f2. Latest consensus state: current_term: 3 leader_uuid: "ccba1e41742b40948281c089a072e8f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1379406ff942b883b7fca5a80c5873" member_type: VOTER last_known_addr { host: "127.6.117.254" port: 44289 } } peers { permanent_uuid: "ccba1e41742b40948281c089a072e8f2" member_type: VOTER last_known_addr { host: "127.6.117.253" port: 39267 } } peers { permanent_uuid: "583a10c367c944f798b885f9c08a6e49" member_type: VOTER last_known_addr { host: "127.6.117.252" port: 44737 } } }
I20240502 01:58:12.717994 8272 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba1379406ff942b883b7fca5a80c5873 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 01:58:12.718739 8266 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccba1e41742b40948281c089a072e8f2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "ccba1e41742b40948281c089a072e8f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1379406ff942b883b7fca5a80c5873" member_type: VOTER last_known_addr { host: "127.6.117.254" port: 44289 } } peers { permanent_uuid: "ccba1e41742b40948281c089a072e8f2" member_type: VOTER last_known_addr { host: "127.6.117.253" port: 39267 } } peers { permanent_uuid: "583a10c367c944f798b885f9c08a6e49" member_type: VOTER last_known_addr { host: "127.6.117.252" port: 44737 } } }
I20240502 01:58:12.718868 8266 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccba1e41742b40948281c089a072e8f2 [sys.catalog]: This master's current role is: LEADER
I20240502 01:58:12.719209 8272 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba1379406ff942b883b7fca5a80c5873 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "ccba1e41742b40948281c089a072e8f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1379406ff942b883b7fca5a80c5873" member_type: VOTER last_known_addr { host: "127.6.117.254" port: 44289 } } peers { permanent_uuid: "ccba1e41742b40948281c089a072e8f2" member_type: VOTER last_known_addr { host: "127.6.117.253" port: 39267 } } peers { permanent_uuid: "583a10c367c944f798b885f9c08a6e49" member_type: VOTER last_known_addr { host: "127.6.117.252" port: 44737 } } }
I20240502 01:58:12.719300 8272 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba1379406ff942b883b7fca5a80c5873 [sys.catalog]: This master's current role is: FOLLOWER
W20240502 01:58:12.719684 7793 consensus_peers.cc:485] T 00000000000000000000000000000000 P ccba1e41742b40948281c089a072e8f2 -> Peer 583a10c367c944f798b885f9c08a6e49 (127.6.117.252:44737): Couldn't send request to peer 583a10c367c944f798b885f9c08a6e49. Status: Network error: Client connection negotiation failed: client connection to 127.6.117.252:44737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:12.838480 7951 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:58:12.838589 7951 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:58:13.337496 6615 tablet_server.cc:178] TabletServer@127.6.117.193:0 shutting down...
I20240502 01:58:13.341289 6615 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:58:13.341516 6615 tablet_replica.cc:330] T ff9b10bd4e4c468abf696ea0a992d748 P a29ce75cc1f74a63a6c8f1d090e4127e: stopping tablet replica
I20240502 01:58:13.341671 6615 raft_consensus.cc:2229] T ff9b10bd4e4c468abf696ea0a992d748 P a29ce75cc1f74a63a6c8f1d090e4127e [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:58:13.341810 6615 raft_consensus.cc:2258] T ff9b10bd4e4c468abf696ea0a992d748 P a29ce75cc1f74a63a6c8f1d090e4127e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:13.354429 6615 tablet_server.cc:194] TabletServer@127.6.117.193:0 shutdown complete.
I20240502 01:58:13.356666 6615 tablet_server.cc:178] TabletServer@127.6.117.194:0 shutting down...
I20240502 01:58:13.360388 6615 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:58:13.360563 6615 tablet_replica.cc:330] T f3aab46168b743f690b431c176f9ee47 P 0314edf5ba0d486bafc4ace92679c4e3: stopping tablet replica
I20240502 01:58:13.360657 6615 raft_consensus.cc:2229] T f3aab46168b743f690b431c176f9ee47 P 0314edf5ba0d486bafc4ace92679c4e3 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:58:13.360765 6615 raft_consensus.cc:2258] T f3aab46168b743f690b431c176f9ee47 P 0314edf5ba0d486bafc4ace92679c4e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:13.373209 6615 tablet_server.cc:194] TabletServer@127.6.117.194:0 shutdown complete.
I20240502 01:58:13.375209 6615 tablet_server.cc:178] TabletServer@127.6.117.195:0 shutting down...
I20240502 01:58:13.378966 6615 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:58:13.379195 6615 tablet_replica.cc:330] T f2570e6e4f294268a8ef862a8f22dc90 P 6cc7355d5e4f4545a69798ecd7f9cb88: stopping tablet replica
I20240502 01:58:13.379292 6615 raft_consensus.cc:2229] T f2570e6e4f294268a8ef862a8f22dc90 P 6cc7355d5e4f4545a69798ecd7f9cb88 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:58:13.379400 6615 raft_consensus.cc:2258] T f2570e6e4f294268a8ef862a8f22dc90 P 6cc7355d5e4f4545a69798ecd7f9cb88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:13.391839 6615 tablet_server.cc:194] TabletServer@127.6.117.195:0 shutdown complete.
I20240502 01:58:13.393919 6615 master.cc:537] Master@127.6.117.254:44289 shutting down...
I20240502 01:58:13.397078 6615 raft_consensus.cc:2229] T 00000000000000000000000000000000 P ba1379406ff942b883b7fca5a80c5873 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:13.397236 6615 raft_consensus.cc:2258] T 00000000000000000000000000000000 P ba1379406ff942b883b7fca5a80c5873 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:13.397286 6615 tablet_replica.cc:330] T 00000000000000000000000000000000 P ba1379406ff942b883b7fca5a80c5873: stopping tablet replica
I20240502 01:58:13.410823 6615 master.cc:552] Master@127.6.117.254:44289 shutdown complete.
I20240502 01:58:13.413589 6615 master.cc:537] Master@127.6.117.253:39267 shutting down...
I20240502 01:58:13.416965 6615 raft_consensus.cc:2229] T 00000000000000000000000000000000 P ccba1e41742b40948281c089a072e8f2 [term 3 LEADER]: Raft consensus shutting down.
I20240502 01:58:13.417202 6615 raft_consensus.cc:2258] T 00000000000000000000000000000000 P ccba1e41742b40948281c089a072e8f2 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:13.417259 6615 tablet_replica.cc:330] T 00000000000000000000000000000000 P ccba1e41742b40948281c089a072e8f2: stopping tablet replica
I20240502 01:58:13.429809 6615 master.cc:552] Master@127.6.117.253:39267 shutdown complete.
[ OK ] AutoRebalancerTest.NextLeaderResumesAutoRebalancing (6723 ms)
[ RUN ] AutoRebalancerTest.MovesScheduledIfAddTserver
I20240502 01:58:13.436900 6615 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.6.117.254:36539
I20240502 01:58:13.437214 6615 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:58:13.439339 8280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:13.439779 8281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:13.440212 8283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:58:13.440593 6615 server_base.cc:1034] running on GCE node
I20240502 01:58:13.440742 6615 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:58:13.440779 6615 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:58:13.440793 6615 hybrid_clock.cc:648] HybridClock initialized: now 1714615093440792 us; error 0 us; skew 500 ppm
I20240502 01:58:13.446303 6615 webserver.cc:460] Webserver started at http://127.6.117.254:42335/ using document root <none> and password file <none>
I20240502 01:58:13.446460 6615 fs_manager.cc:359] Metadata directory not provided
I20240502 01:58:13.446511 6615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:58:13.446599 6615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:58:13.446987 6615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714615076189642-6615-0/minicluster-data/master-0-root/instance:
uuid: "6ac6781202f74b91be784b5f5196b90f"
format_stamp: "Formatted at 2024-05-02 01:58:13 on dist-test-slave-89cm"
I20240502 01:58:13.448480 6615 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240502 01:58:13.449501 8288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:13.449784 6615 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:13.449847 6615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714615076189642-6615-0/minicluster-data/master-0-root
uuid: "6ac6781202f74b91be784b5f5196b90f"
format_stamp: "Formatted at 2024-05-02 01:58:13 on dist-test-slave-89cm"
I20240502 01:58:13.449904 6615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714615076189642-6615-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714615076189642-6615-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714615076189642-6615-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:58:13.459450 6615 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:58:13.459777 6615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:58:13.463419 6615 rpc_server.cc:307] RPC server started. Bound to: 127.6.117.254:36539
I20240502 01:58:13.465813 8340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.117.254:36539 every 8 connection(s)
I20240502 01:58:13.466252 8341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:13.467983 8341 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ac6781202f74b91be784b5f5196b90f: Bootstrap starting.
I20240502 01:58:13.468775 8341 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ac6781202f74b91be784b5f5196b90f: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:13.469751 8341 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ac6781202f74b91be784b5f5196b90f: No bootstrap required, opened a new log
I20240502 01:58:13.470109 8341 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ac6781202f74b91be784b5f5196b90f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ac6781202f74b91be784b5f5196b90f" member_type: VOTER }
I20240502 01:58:13.470198 8341 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ac6781202f74b91be784b5f5196b90f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:13.470229 8341 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ac6781202f74b91be784b5f5196b90f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ac6781202f74b91be784b5f5196b90f, State: Initialized, Role: FOLLOWER
I20240502 01:58:13.470351 8341 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ac6781202f74b91be784b5f5196b90f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ac6781202f74b91be784b5f5196b90f" member_type: VOTER }
I20240502 01:58:13.470423 8341 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ac6781202f74b91be784b5f5196b90f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:58:13.470454 8341 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ac6781202f74b91be784b5f5196b90f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:58:13.470499 8341 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ac6781202f74b91be784b5f5196b90f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:13.471140 8341 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ac6781202f74b91be784b5f5196b90f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ac6781202f74b91be784b5f5196b90f" member_type: VOTER }
I20240502 01:58:13.471253 8341 leader_election.cc:302] T 00000000000000000000000000000000 P 6ac6781202f74b91be784b5f5196b90f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ac6781202f74b91be784b5f5196b90f; no voters:
I20240502 01:58:13.471424 8341 leader_election.cc:288] T 00000000000000000000000000000000 P 6ac6781202f74b91be784b5f5196b90f [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:58:13.471601 8344 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ac6781202f74b91be784b5f5196b90f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:13.471779 8344 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ac6781202f74b91be784b5f5196b90f [term 1 LEADER]: Becoming Leader. State: Replica: 6ac6781202f74b91be784b5f5196b90f, State: Running, Role: LEADER
I20240502 01:58:13.471794 8341 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ac6781202f74b91be784b5f5196b90f [sys.catalog]: configured and running, proceeding with master startup.
I20240502 01:58:13.471951 8344 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ac6781202f74b91be784b5f5196b90f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ac6781202f74b91be784b5f5196b90f" member_type: VOTER }
I20240502 01:58:13.472357 8345 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ac6781202f74b91be784b5f5196b90f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6ac6781202f74b91be784b5f5196b90f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ac6781202f74b91be784b5f5196b90f" member_type: VOTER } }
I20240502 01:58:13.472434 8345 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ac6781202f74b91be784b5f5196b90f [sys.catalog]: This master's current role is: LEADER
I20240502 01:58:13.472393 8346 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ac6781202f74b91be784b5f5196b90f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ac6781202f74b91be784b5f5196b90f. Latest consensus state: current_term: 1 leader_uuid: "6ac6781202f74b91be784b5f5196b90f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ac6781202f74b91be784b5f5196b90f" member_type: VOTER } }
I20240502 01:58:13.472483 8346 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ac6781202f74b91be784b5f5196b90f [sys.catalog]: This master's current role is: LEADER
I20240502 01:58:13.472682 8350 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 01:58:13.473575 8350 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 01:58:13.473734 6615 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 01:58:13.475416 8350 catalog_manager.cc:1347] Generated new cluster ID: 7c52837f0f7747b7971d6b9bbc6415e2
I20240502 01:58:13.475461 8350 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 01:58:13.499976 8350 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 01:58:13.500584 8350 catalog_manager.cc:1504] Loading token signing keys...
I20240502 01:58:13.505791 8350 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 6ac6781202f74b91be784b5f5196b90f: Generated new TSK 0
I20240502 01:58:13.505970 8350 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 01:58:13.538452 6615 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:58:13.540629 8362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:13.541033 8365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:13.541401 8363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:58:13.541782 6615 server_base.cc:1034] running on GCE node
I20240502 01:58:13.541941 6615 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:58:13.541990 6615 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:58:13.542017 6615 hybrid_clock.cc:648] HybridClock initialized: now 1714615093542017 us; error 0 us; skew 500 ppm
I20240502 01:58:13.542989 6615 webserver.cc:460] Webserver started at http://127.6.117.193:36391/ using document root <none> and password file <none>
I20240502 01:58:13.543133 6615 fs_manager.cc:359] Metadata directory not provided
I20240502 01:58:13.543181 6615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:58:13.543260 6615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:58:13.543681 6615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714615076189642-6615-0/minicluster-data/ts-0-root/instance:
uuid: "7a9001351fe84a898b8544ef84aab5b9"
format_stamp: "Formatted at 2024-05-02 01:58:13 on dist-test-slave-89cm"
I20240502 01:58:13.545104 6615 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240502 01:58:13.545940 8370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:13.546165 6615 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:13.546227 6615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714615076189642-6615-0/minicluster-data/ts-0-root
uuid: "7a9001351fe84a898b8544ef84aab5b9"
format_stamp: "Formatted at 2024-05-02 01:58:13 on dist-test-slave-89cm"
I20240502 01:58:13.546294 6615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714615076189642-6615-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714615076189642-6615-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714615076189642-6615-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:58:13.558470 6615 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:58:13.558866 6615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:58:13.559151 6615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:58:13.559605 6615 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:58:13.559645 6615 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:13.559680 6615 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:58:13.559706 6615 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:13.563601 6615 rpc_server.cc:307] RPC server started. Bound to: 127.6.117.193:43793
I20240502 01:58:13.563630 8433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.117.193:43793 every 8 connection(s)
I20240502 01:58:13.564481 6615 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:58:13.566447 8438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:13.566761 8439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:58:13.567195 6615 server_base.cc:1034] running on GCE node
W20240502 01:58:13.567466 8441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:58:13.567736 6615 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:58:13.567795 6615 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:58:13.567817 6615 hybrid_clock.cc:648] HybridClock initialized: now 1714615093567817 us; error 0 us; skew 500 ppm
I20240502 01:58:13.568761 6615 webserver.cc:460] Webserver started at http://127.6.117.194:37721/ using document root <none> and password file <none>
I20240502 01:58:13.568915 6615 fs_manager.cc:359] Metadata directory not provided
I20240502 01:58:13.568975 6615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:58:13.568992 8434 heartbeater.cc:347] Connected to a master server at 127.6.117.254:36539
I20240502 01:58:13.569051 6615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:58:13.569080 8434 heartbeater.cc:464] Registering TS with master...
I20240502 01:58:13.569314 8434 heartbeater.cc:511] Master 127.6.117.254:36539 requested a full tablet report, sending...
I20240502 01:58:13.569525 6615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714615076189642-6615-0/minicluster-data/ts-1-root/instance:
uuid: "ca98fb22a15140c6a26cca8192d0b6ba"
format_stamp: "Formatted at 2024-05-02 01:58:13 on dist-test-slave-89cm"
I20240502 01:58:13.569929 8305 ts_manager.cc:194] Registered new tserver with Master: 7a9001351fe84a898b8544ef84aab5b9 (127.6.117.193:43793)
I20240502 01:58:13.570648 8305 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45754
I20240502 01:58:13.570950 6615 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240502 01:58:13.571951 8446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:13.572165 6615 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:13.572227 6615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714615076189642-6615-0/minicluster-data/ts-1-root
uuid: "ca98fb22a15140c6a26cca8192d0b6ba"
format_stamp: "Formatted at 2024-05-02 01:58:13 on dist-test-slave-89cm"
I20240502 01:58:13.572309 6615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714615076189642-6615-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714615076189642-6615-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714615076189642-6615-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:58:13.589619 6615 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:58:13.590008 6615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:58:13.590302 6615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:58:13.590785 6615 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:58:13.590829 6615 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:13.590873 6615 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:58:13.590902 6615 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:13.594806 6615 rpc_server.cc:307] RPC server started. Bound to: 127.6.117.194:33585
I20240502 01:58:13.594892 8509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.117.194:33585 every 8 connection(s)
I20240502 01:58:13.595579 6615 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:58:13.597498 8514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:13.597764 8513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:58:13.598217 6615 server_base.cc:1034] running on GCE node
W20240502 01:58:13.598515 8516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:58:13.598709 6615 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:58:13.598752 6615 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:58:13.598766 6615 hybrid_clock.cc:648] HybridClock initialized: now 1714615093598766 us; error 0 us; skew 500 ppm
I20240502 01:58:13.599593 6615 webserver.cc:460] Webserver started at http://127.6.117.195:41977/ using document root <none> and password file <none>
I20240502 01:58:13.599750 6615 fs_manager.cc:359] Metadata directory not provided
I20240502 01:58:13.599802 6615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:58:13.599871 6615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:58:13.600020 8510 heartbeater.cc:347] Connected to a master server at 127.6.117.254:36539
I20240502 01:58:13.600112 8510 heartbeater.cc:464] Registering TS with master...
I20240502 01:58:13.600365 8510 heartbeater.cc:511] Master 127.6.117.254:36539 requested a full tablet report, sending...
I20240502 01:58:13.600353 6615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714615076189642-6615-0/minicluster-data/ts-2-root/instance:
uuid: "d87d866f3cbf4c14b4921a68f8779d54"
format_stamp: "Formatted at 2024-05-02 01:58:13 on dist-test-slave-89cm"
I20240502 01:58:13.601033 8305 ts_manager.cc:194] Registered new tserver with Master: ca98fb22a15140c6a26cca8192d0b6ba (127.6.117.194:33585)
I20240502 01:58:13.601559 8305 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45758
I20240502 01:58:13.601919 6615 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240502 01:58:13.602813 8521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:13.603022 6615 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240502 01:58:13.603085 6615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714615076189642-6615-0/minicluster-data/ts-2-root
uuid: "d87d866f3cbf4c14b4921a68f8779d54"
format_stamp: "Formatted at 2024-05-02 01:58:13 on dist-test-slave-89cm"
I20240502 01:58:13.603152 6615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714615076189642-6615-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714615076189642-6615-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714615076189642-6615-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:58:13.620887 6615 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:58:13.621281 6615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:58:13.621578 6615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:58:13.622097 6615 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:58:13.622138 6615 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:13.622182 6615 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:58:13.622212 6615 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:13.626355 6615 rpc_server.cc:307] RPC server started. Bound to: 127.6.117.195:41451
I20240502 01:58:13.626410 8584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.117.195:41451 every 8 connection(s)
I20240502 01:58:13.630714 8585 heartbeater.cc:347] Connected to a master server at 127.6.117.254:36539
I20240502 01:58:13.630821 8585 heartbeater.cc:464] Registering TS with master...
I20240502 01:58:13.631047 8585 heartbeater.cc:511] Master 127.6.117.254:36539 requested a full tablet report, sending...
I20240502 01:58:13.631647 8305 ts_manager.cc:194] Registered new tserver with Master: d87d866f3cbf4c14b4921a68f8779d54 (127.6.117.195:41451)
I20240502 01:58:13.632169 8305 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45768
I20240502 01:58:13.632323 6615 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.005586691s
I20240502 01:58:14.571600 8434 heartbeater.cc:503] Master 127.6.117.254:36539 was elected leader, sending a full tablet report...
I20240502 01:58:14.602440 8510 heartbeater.cc:503] Master 127.6.117.254:36539 was elected leader, sending a full tablet report...
I20240502 01:58:14.633057 8585 heartbeater.cc:503] Master 127.6.117.254:36539 was elected leader, sending a full tablet report...
I20240502 01:58:14.658646 6615 test_util.cc:268] Using random seed: -274413998
I20240502 01:58:14.666615 8305 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45772:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377?""\004\001\000\377\377\377?"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 01:58:14.667090 8305 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 01:58:14.677264 8398 tablet_service.cc:1467] Processing CreateTablet for tablet b404009b1e4e4be89b787476201c68a9 (DEFAULT_TABLE table=test-workload [id=a959bafba1434effa4b51918619ee5c1]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240502 01:58:14.678614 8549 tablet_service.cc:1467] Processing CreateTablet for tablet b404009b1e4e4be89b787476201c68a9 (DEFAULT_TABLE table=test-workload [id=a959bafba1434effa4b51918619ee5c1]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240502 01:58:14.680166 8398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b404009b1e4e4be89b787476201c68a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:14.680467 8549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b404009b1e4e4be89b787476201c68a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:14.678660 8474 tablet_service.cc:1467] Processing CreateTablet for tablet b404009b1e4e4be89b787476201c68a9 (DEFAULT_TABLE table=test-workload [id=a959bafba1434effa4b51918619ee5c1]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240502 01:58:14.683033 8474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b404009b1e4e4be89b787476201c68a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:14.683084 8605 tablet_bootstrap.cc:492] T b404009b1e4e4be89b787476201c68a9 P 7a9001351fe84a898b8544ef84aab5b9: Bootstrap starting.
I20240502 01:58:14.684177 8605 tablet_bootstrap.cc:654] T b404009b1e4e4be89b787476201c68a9 P 7a9001351fe84a898b8544ef84aab5b9: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:14.684970 8606 tablet_bootstrap.cc:492] T b404009b1e4e4be89b787476201c68a9 P d87d866f3cbf4c14b4921a68f8779d54: Bootstrap starting.
I20240502 01:58:14.685379 8605 tablet_bootstrap.cc:492] T b404009b1e4e4be89b787476201c68a9 P 7a9001351fe84a898b8544ef84aab5b9: No bootstrap required, opened a new log
I20240502 01:58:14.685469 8608 tablet_bootstrap.cc:492] T b404009b1e4e4be89b787476201c68a9 P ca98fb22a15140c6a26cca8192d0b6ba: Bootstrap starting.
I20240502 01:58:14.685564 8605 ts_tablet_manager.cc:1367] T b404009b1e4e4be89b787476201c68a9 P 7a9001351fe84a898b8544ef84aab5b9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:58:14.678956 8473 tablet_service.cc:1467] Processing CreateTablet for tablet 21b7916993614f6d98c2f7460920a827 (DEFAULT_TABLE table=test-workload [id=a959bafba1434effa4b51918619ee5c1]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240502 01:58:14.685896 8473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21b7916993614f6d98c2f7460920a827. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:14.685981 8606 tablet_bootstrap.cc:654] T b404009b1e4e4be89b787476201c68a9 P d87d866f3cbf4c14b4921a68f8779d54: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:14.686336 8608 tablet_bootstrap.cc:654] T b404009b1e4e4be89b787476201c68a9 P ca98fb22a15140c6a26cca8192d0b6ba: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:14.686661 8605 raft_consensus.cc:348] T b404009b1e4e4be89b787476201c68a9 P 7a9001351fe84a898b8544ef84aab5b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca98fb22a15140c6a26cca8192d0b6ba" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 33585 } } peers { permanent_uuid: "7a9001351fe84a898b8544ef84aab5b9" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 43793 } } peers { permanent_uuid: "d87d866f3cbf4c14b4921a68f8779d54" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 41451 } }
I20240502 01:58:14.686866 8605 raft_consensus.cc:374] T b404009b1e4e4be89b787476201c68a9 P 7a9001351fe84a898b8544ef84aab5b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:14.686949 8605 raft_consensus.cc:729] T b404009b1e4e4be89b787476201c68a9 P 7a9001351fe84a898b8544ef84aab5b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a9001351fe84a898b8544ef84aab5b9, State: Initialized, Role: FOLLOWER
I20240502 01:58:14.687139 8605 consensus_queue.cc:260] T b404009b1e4e4be89b787476201c68a9 P 7a9001351fe84a898b8544ef84aab5b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca98fb22a15140c6a26cca8192d0b6ba" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 33585 } } peers { permanent_uuid: "7a9001351fe84a898b8544ef84aab5b9" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 43793 } } peers { permanent_uuid: "d87d866f3cbf4c14b4921a68f8779d54" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 41451 } }
I20240502 01:58:14.687538 8605 ts_tablet_manager.cc:1398] T b404009b1e4e4be89b787476201c68a9 P 7a9001351fe84a898b8544ef84aab5b9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:14.679248 8397 tablet_service.cc:1467] Processing CreateTablet for tablet 21b7916993614f6d98c2f7460920a827 (DEFAULT_TABLE table=test-workload [id=a959bafba1434effa4b51918619ee5c1]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240502 01:58:14.679823 8548 tablet_service.cc:1467] Processing CreateTablet for tablet 21b7916993614f6d98c2f7460920a827 (DEFAULT_TABLE table=test-workload [id=a959bafba1434effa4b51918619ee5c1]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240502 01:58:14.688289 8397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21b7916993614f6d98c2f7460920a827. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:14.688318 8548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21b7916993614f6d98c2f7460920a827. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:14.688588 8608 tablet_bootstrap.cc:492] T b404009b1e4e4be89b787476201c68a9 P ca98fb22a15140c6a26cca8192d0b6ba: No bootstrap required, opened a new log
I20240502 01:58:14.688664 8608 ts_tablet_manager.cc:1367] T b404009b1e4e4be89b787476201c68a9 P ca98fb22a15140c6a26cca8192d0b6ba: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:58:14.689062 8608 raft_consensus.cc:348] T b404009b1e4e4be89b787476201c68a9 P ca98fb22a15140c6a26cca8192d0b6ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca98fb22a15140c6a26cca8192d0b6ba" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 33585 } } peers { permanent_uuid: "7a9001351fe84a898b8544ef84aab5b9" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 43793 } } peers { permanent_uuid: "d87d866f3cbf4c14b4921a68f8779d54" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 41451 } }
I20240502 01:58:14.689154 8608 raft_consensus.cc:374] T b404009b1e4e4be89b787476201c68a9 P ca98fb22a15140c6a26cca8192d0b6ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:14.689181 8608 raft_consensus.cc:729] T b404009b1e4e4be89b787476201c68a9 P ca98fb22a15140c6a26cca8192d0b6ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca98fb22a15140c6a26cca8192d0b6ba, State: Initialized, Role: FOLLOWER
I20240502 01:58:14.689301 8608 consensus_queue.cc:260] T b404009b1e4e4be89b787476201c68a9 P ca98fb22a15140c6a26cca8192d0b6ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca98fb22a15140c6a26cca8192d0b6ba" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 33585 } } peers { permanent_uuid: "7a9001351fe84a898b8544ef84aab5b9" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 43793 } } peers { permanent_uuid: "d87d866f3cbf4c14b4921a68f8779d54" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 41451 } }
I20240502 01:58:14.689630 8608 ts_tablet_manager.cc:1398] T b404009b1e4e4be89b787476201c68a9 P ca98fb22a15140c6a26cca8192d0b6ba: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:58:14.689781 8608 tablet_bootstrap.cc:492] T 21b7916993614f6d98c2f7460920a827 P ca98fb22a15140c6a26cca8192d0b6ba: Bootstrap starting.
I20240502 01:58:14.690467 8605 tablet_bootstrap.cc:492] T 21b7916993614f6d98c2f7460920a827 P 7a9001351fe84a898b8544ef84aab5b9: Bootstrap starting.
I20240502 01:58:14.690614 8608 tablet_bootstrap.cc:654] T 21b7916993614f6d98c2f7460920a827 P ca98fb22a15140c6a26cca8192d0b6ba: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:14.691340 8605 tablet_bootstrap.cc:654] T 21b7916993614f6d98c2f7460920a827 P 7a9001351fe84a898b8544ef84aab5b9: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:14.692976 8606 tablet_bootstrap.cc:492] T b404009b1e4e4be89b787476201c68a9 P d87d866f3cbf4c14b4921a68f8779d54: No bootstrap required, opened a new log
I20240502 01:58:14.692991 8605 tablet_bootstrap.cc:492] T 21b7916993614f6d98c2f7460920a827 P 7a9001351fe84a898b8544ef84aab5b9: No bootstrap required, opened a new log
I20240502 01:58:14.693099 8606 ts_tablet_manager.cc:1367] T b404009b1e4e4be89b787476201c68a9 P d87d866f3cbf4c14b4921a68f8779d54: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240502 01:58:14.693166 8605 ts_tablet_manager.cc:1367] T 21b7916993614f6d98c2f7460920a827 P 7a9001351fe84a898b8544ef84aab5b9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:58:14.693543 8606 raft_consensus.cc:348] T b404009b1e4e4be89b787476201c68a9 P d87d866f3cbf4c14b4921a68f8779d54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca98fb22a15140c6a26cca8192d0b6ba" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 33585 } } peers { permanent_uuid: "7a9001351fe84a898b8544ef84aab5b9" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 43793 } } peers { permanent_uuid: "d87d866f3cbf4c14b4921a68f8779d54" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 41451 } }
I20240502 01:58:14.693667 8606 raft_consensus.cc:374] T b404009b1e4e4be89b787476201c68a9 P d87d866f3cbf4c14b4921a68f8779d54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:14.693709 8606 raft_consensus.cc:729] T b404009b1e4e4be89b787476201c68a9 P d87d866f3cbf4c14b4921a68f8779d54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d87d866f3cbf4c14b4921a68f8779d54, State: Initialized, Role: FOLLOWER
I20240502 01:58:14.693796 8608 tablet_bootstrap.cc:492] T 21b7916993614f6d98c2f7460920a827 P ca98fb22a15140c6a26cca8192d0b6ba: No bootstrap required, opened a new log
I20240502 01:58:14.693910 8608 ts_tablet_manager.cc:1367] T 21b7916993614f6d98c2f7460920a827 P ca98fb22a15140c6a26cca8192d0b6ba: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240502 01:58:14.693881 8606 consensus_queue.cc:260] T b404009b1e4e4be89b787476201c68a9 P d87d866f3cbf4c14b4921a68f8779d54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca98fb22a15140c6a26cca8192d0b6ba" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 33585 } } peers { permanent_uuid: "7a9001351fe84a898b8544ef84aab5b9" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 43793 } } peers { permanent_uuid: "d87d866f3cbf4c14b4921a68f8779d54" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 41451 } }
I20240502 01:58:14.694245 8606 ts_tablet_manager.cc:1398] T b404009b1e4e4be89b787476201c68a9 P d87d866f3cbf4c14b4921a68f8779d54: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:14.694303 8608 raft_consensus.cc:348] T 21b7916993614f6d98c2f7460920a827 P ca98fb22a15140c6a26cca8192d0b6ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a9001351fe84a898b8544ef84aab5b9" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 43793 } } peers { permanent_uuid: "ca98fb22a15140c6a26cca8192d0b6ba" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 33585 } } peers { permanent_uuid: "d87d866f3cbf4c14b4921a68f8779d54" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 41451 } }
I20240502 01:58:14.694399 8608 raft_consensus.cc:374] T 21b7916993614f6d98c2f7460920a827 P ca98fb22a15140c6a26cca8192d0b6ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:14.694433 8606 tablet_bootstrap.cc:492] T 21b7916993614f6d98c2f7460920a827 P d87d866f3cbf4c14b4921a68f8779d54: Bootstrap starting.
I20240502 01:58:14.693689 8605 raft_consensus.cc:348] T 21b7916993614f6d98c2f7460920a827 P 7a9001351fe84a898b8544ef84aab5b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a9001351fe84a898b8544ef84aab5b9" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 43793 } } peers { permanent_uuid: "ca98fb22a15140c6a26cca8192d0b6ba" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 33585 } } peers { permanent_uuid: "d87d866f3cbf4c14b4921a68f8779d54" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 41451 } }
I20240502 01:58:14.694515 8605 raft_consensus.cc:374] T 21b7916993614f6d98c2f7460920a827 P 7a9001351fe84a898b8544ef84aab5b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:14.694551 8605 raft_consensus.cc:729] T 21b7916993614f6d98c2f7460920a827 P 7a9001351fe84a898b8544ef84aab5b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a9001351fe84a898b8544ef84aab5b9, State: Initialized, Role: FOLLOWER
I20240502 01:58:14.694658 8605 consensus_queue.cc:260] T 21b7916993614f6d98c2f7460920a827 P 7a9001351fe84a898b8544ef84aab5b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a9001351fe84a898b8544ef84aab5b9" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 43793 } } peers { permanent_uuid: "ca98fb22a15140c6a26cca8192d0b6ba" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 33585 } } peers { permanent_uuid: "d87d866f3cbf4c14b4921a68f8779d54" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 41451 } }
I20240502 01:58:14.694898 8605 ts_tablet_manager.cc:1398] T 21b7916993614f6d98c2f7460920a827 P 7a9001351fe84a898b8544ef84aab5b9: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:58:14.695322 8606 tablet_bootstrap.cc:654] T 21b7916993614f6d98c2f7460920a827 P d87d866f3cbf4c14b4921a68f8779d54: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:14.694429 8608 raft_consensus.cc:729] T 21b7916993614f6d98c2f7460920a827 P ca98fb22a15140c6a26cca8192d0b6ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca98fb22a15140c6a26cca8192d0b6ba, State: Initialized, Role: FOLLOWER
I20240502 01:58:14.696003 8608 consensus_queue.cc:260] T 21b7916993614f6d98c2f7460920a827 P ca98fb22a15140c6a26cca8192d0b6ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a9001351fe84a898b8544ef84aab5b9" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 43793 } } peers { permanent_uuid: "ca98fb22a15140c6a26cca8192d0b6ba" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 33585 } } peers { permanent_uuid: "d87d866f3cbf4c14b4921a68f8779d54" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 41451 } }
I20240502 01:58:14.696291 8608 ts_tablet_manager.cc:1398] T 21b7916993614f6d98c2f7460920a827 P ca98fb22a15140c6a26cca8192d0b6ba: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:58:14.696849 8606 tablet_bootstrap.cc:492] T 21b7916993614f6d98c2f7460920a827 P d87d866f3cbf4c14b4921a68f8779d54: No bootstrap required, opened a new log
I20240502 01:58:14.697018 8606 ts_tablet_manager.cc:1367] T 21b7916993614f6d98c2f7460920a827 P d87d866f3cbf4c14b4921a68f8779d54: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:58:14.697451 8606 raft_consensus.cc:348] T 21b7916993614f6d98c2f7460920a827 P d87d866f3cbf4c14b4921a68f8779d54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a9001351fe84a898b8544ef84aab5b9" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 43793 } } peers { permanent_uuid: "ca98fb22a15140c6a26cca8192d0b6ba" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 33585 } } peers { permanent_uuid: "d87d866f3cbf4c14b4921a68f8779d54" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 41451 } }
I20240502 01:58:14.697613 8606 raft_consensus.cc:374] T 21b7916993614f6d98c2f7460920a827 P d87d866f3cbf4c14b4921a68f8779d54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:14.697695 8606 raft_consensus.cc:729] T 21b7916993614f6d98c2f7460920a827 P d87d866f3cbf4c14b4921a68f8779d54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d87d866f3cbf4c14b4921a68f8779d54, State: Initialized, Role: FOLLOWER
I20240502 01:58:14.697868 8606 consensus_queue.cc:260] T 21b7916993614f6d98c2f7460920a827 P d87d866f3cbf4c14b4921a68f8779d54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a9001351fe84a898b8544ef84aab5b9" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 43793 } } peers { permanent_uuid: "ca98fb22a15140c6a26cca8192d0b6ba" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 33585 } } peers { permanent_uuid: "d87d866f3cbf4c14b4921a68f8779d54" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 41451 } }
I20240502 01:58:14.698169 8606 ts_tablet_manager.cc:1398] T 21b7916993614f6d98c2f7460920a827 P d87d866f3cbf4c14b4921a68f8779d54: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:14.827349 8611 raft_consensus.cc:482] T b404009b1e4e4be89b787476201c68a9 P 7a9001351fe84a898b8544ef84aab5b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:14.827512 8611 raft_consensus.cc:504] T b404009b1e4e4be89b787476201c68a9 P 7a9001351fe84a898b8544ef84aab5b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca98fb22a15140c6a26cca8192d0b6ba" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 33585 } } peers { permanent_uuid: "7a9001351fe84a898b8544ef84aab5b9" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 43793 } } peers { permanent_uuid: "d87d866f3cbf4c14b4921a68f8779d54" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 41451 } }
I20240502 01:58:14.828084 8611 leader_election.cc:288] T b404009b1e4e4be89b787476201c68a9 P 7a9001351fe84a898b8544ef84aab5b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ca98fb22a15140c6a26cca8192d0b6ba (127.6.117.194:33585), d87d866f3cbf4c14b4921a68f8779d54 (127.6.117.195:41451)
I20240502 01:58:14.832185 8559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b404009b1e4e4be89b787476201c68a9" candidate_uuid: "7a9001351fe84a898b8544ef84aab5b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d87d866f3cbf4c14b4921a68f8779d54" is_pre_election: true
I20240502 01:58:14.832190 8484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b404009b1e4e4be89b787476201c68a9" candidate_uuid: "7a9001351fe84a898b8544ef84aab5b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca98fb22a15140c6a26cca8192d0b6ba" is_pre_election: true
I20240502 01:58:14.832410 8559 raft_consensus.cc:2454] T b404009b1e4e4be89b787476201c68a9 P d87d866f3cbf4c14b4921a68f8779d54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9001351fe84a898b8544ef84aab5b9 in term 0.
I20240502 01:58:14.832441 8484 raft_consensus.cc:2454] T b404009b1e4e4be89b787476201c68a9 P ca98fb22a15140c6a26cca8192d0b6ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9001351fe84a898b8544ef84aab5b9 in term 0.
I20240502 01:58:14.832695 8372 leader_election.cc:302] T b404009b1e4e4be89b787476201c68a9 P 7a9001351fe84a898b8544ef84aab5b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a9001351fe84a898b8544ef84aab5b9, d87d866f3cbf4c14b4921a68f8779d54; no voters:
I20240502 01:58:14.832865 8611 raft_consensus.cc:2781] T b404009b1e4e4be89b787476201c68a9 P 7a9001351fe84a898b8544ef84aab5b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:14.832928 8611 raft_consensus.cc:482] T b404009b1e4e4be89b787476201c68a9 P 7a9001351fe84a898b8544ef84aab5b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:14.832978 8611 raft_consensus.cc:3037] T b404009b1e4e4be89b787476201c68a9 P 7a9001351fe84a898b8544ef84aab5b9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:14.833798 8611 raft_consensus.cc:504] T b404009b1e4e4be89b787476201c68a9 P 7a9001351fe84a898b8544ef84aab5b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca98fb22a15140c6a26cca8192d0b6ba" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 33585 } } peers { permanent_uuid: "7a9001351fe84a898b8544ef84aab5b9" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 43793 } } peers { permanent_uuid: "d87d866f3cbf4c14b4921a68f8779d54" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 41451 } }
I20240502 01:58:14.834117 8611 leader_election.cc:288] T b404009b1e4e4be89b787476201c68a9 P 7a9001351fe84a898b8544ef84aab5b9 [CANDIDATE]: Term 1 election: Requested vote from peers ca98fb22a15140c6a26cca8192d0b6ba (127.6.117.194:33585), d87d866f3cbf4c14b4921a68f8779d54 (127.6.117.195:41451)
I20240502 01:58:14.834415 8484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b404009b1e4e4be89b787476201c68a9" candidate_uuid: "7a9001351fe84a898b8544ef84aab5b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca98fb22a15140c6a26cca8192d0b6ba"
I20240502 01:58:14.834412 8559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b404009b1e4e4be89b787476201c68a9" candidate_uuid: "7a9001351fe84a898b8544ef84aab5b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d87d866f3cbf4c14b4921a68f8779d54"
I20240502 01:58:14.834535 8484 raft_consensus.cc:3037] T b404009b1e4e4be89b787476201c68a9 P ca98fb22a15140c6a26cca8192d0b6ba [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:14.834571 8559 raft_consensus.cc:3037] T b404009b1e4e4be89b787476201c68a9 P d87d866f3cbf4c14b4921a68f8779d54 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:14.835333 8484 raft_consensus.cc:2454] T b404009b1e4e4be89b787476201c68a9 P ca98fb22a15140c6a26cca8192d0b6ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9001351fe84a898b8544ef84aab5b9 in term 1.
I20240502 01:58:14.835391 8559 raft_consensus.cc:2454] T b404009b1e4e4be89b787476201c68a9 P d87d866f3cbf4c14b4921a68f8779d54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9001351fe84a898b8544ef84aab5b9 in term 1.
I20240502 01:58:14.835602 8373 leader_election.cc:302] T b404009b1e4e4be89b787476201c68a9 P 7a9001351fe84a898b8544ef84aab5b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a9001351fe84a898b8544ef84aab5b9, ca98fb22a15140c6a26cca8192d0b6ba; no voters:
I20240502 01:58:14.835743 8611 raft_consensus.cc:2781] T b404009b1e4e4be89b787476201c68a9 P 7a9001351fe84a898b8544ef84aab5b9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:14.835944 8611 raft_consensus.cc:686] T b404009b1e4e4be89b787476201c68a9 P 7a9001351fe84a898b8544ef84aab5b9 [term 1 LEADER]: Becoming Leader. State: Replica: 7a9001351fe84a898b8544ef84aab5b9, State: Running, Role: LEADER
I20240502 01:58:14.836097 8611 consensus_queue.cc:237] T b404009b1e4e4be89b787476201c68a9 P 7a9001351fe84a898b8544ef84aab5b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca98fb22a15140c6a26cca8192d0b6ba" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 33585 } } peers { permanent_uuid: "7a9001351fe84a898b8544ef84aab5b9" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 43793 } } peers { permanent_uuid: "d87d866f3cbf4c14b4921a68f8779d54" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 41451 } }
I20240502 01:58:14.837541 8304 catalog_manager.cc:5484] T b404009b1e4e4be89b787476201c68a9 P 7a9001351fe84a898b8544ef84aab5b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a9001351fe84a898b8544ef84aab5b9 (127.6.117.193). New cstate: current_term: 1 leader_uuid: "7a9001351fe84a898b8544ef84aab5b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca98fb22a15140c6a26cca8192d0b6ba" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 33585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a9001351fe84a898b8544ef84aab5b9" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 43793 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d87d866f3cbf4c14b4921a68f8779d54" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 41451 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:14.872815 8611 raft_consensus.cc:482] T 21b7916993614f6d98c2f7460920a827 P 7a9001351fe84a898b8544ef84aab5b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:14.872994 8611 raft_consensus.cc:504] T 21b7916993614f6d98c2f7460920a827 P 7a9001351fe84a898b8544ef84aab5b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a9001351fe84a898b8544ef84aab5b9" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 43793 } } peers { permanent_uuid: "ca98fb22a15140c6a26cca8192d0b6ba" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 33585 } } peers { permanent_uuid: "d87d866f3cbf4c14b4921a68f8779d54" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 41451 } }
I20240502 01:58:14.873390 8611 leader_election.cc:288] T 21b7916993614f6d98c2f7460920a827 P 7a9001351fe84a898b8544ef84aab5b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ca98fb22a15140c6a26cca8192d0b6ba (127.6.117.194:33585), d87d866f3cbf4c14b4921a68f8779d54 (127.6.117.195:41451)
I20240502 01:58:14.873768 8559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21b7916993614f6d98c2f7460920a827" candidate_uuid: "7a9001351fe84a898b8544ef84aab5b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d87d866f3cbf4c14b4921a68f8779d54" is_pre_election: true
I20240502 01:58:14.873771 8484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21b7916993614f6d98c2f7460920a827" candidate_uuid: "7a9001351fe84a898b8544ef84aab5b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca98fb22a15140c6a26cca8192d0b6ba" is_pre_election: true
I20240502 01:58:14.873895 8559 raft_consensus.cc:2454] T 21b7916993614f6d98c2f7460920a827 P d87d866f3cbf4c14b4921a68f8779d54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9001351fe84a898b8544ef84aab5b9 in term 0.
I20240502 01:58:14.873921 8484 raft_consensus.cc:2454] T 21b7916993614f6d98c2f7460920a827 P ca98fb22a15140c6a26cca8192d0b6ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9001351fe84a898b8544ef84aab5b9 in term 0.
I20240502 01:58:14.874178 8372 leader_election.cc:302] T 21b7916993614f6d98c2f7460920a827 P 7a9001351fe84a898b8544ef84aab5b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a9001351fe84a898b8544ef84aab5b9, d87d866f3cbf4c14b4921a68f8779d54; no voters:
I20240502 01:58:14.874328 8611 raft_consensus.cc:2781] T 21b7916993614f6d98c2f7460920a827 P 7a9001351fe84a898b8544ef84aab5b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:14.874393 8611 raft_consensus.cc:482] T 21b7916993614f6d98c2f7460920a827 P 7a9001351fe84a898b8544ef84aab5b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:14.874437 8611 raft_consensus.cc:3037] T 21b7916993614f6d98c2f7460920a827 P 7a9001351fe84a898b8544ef84aab5b9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:14.875232 8611 raft_consensus.cc:504] T 21b7916993614f6d98c2f7460920a827 P 7a9001351fe84a898b8544ef84aab5b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a9001351fe84a898b8544ef84aab5b9" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 43793 } } peers { permanent_uuid: "ca98fb22a15140c6a26cca8192d0b6ba" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 33585 } } peers { permanent_uuid: "d87d866f3cbf4c14b4921a68f8779d54" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 41451 } }
I20240502 01:58:14.875555 8611 leader_election.cc:288] T 21b7916993614f6d98c2f7460920a827 P 7a9001351fe84a898b8544ef84aab5b9 [CANDIDATE]: Term 1 election: Requested vote from peers ca98fb22a15140c6a26cca8192d0b6ba (127.6.117.194:33585), d87d866f3cbf4c14b4921a68f8779d54 (127.6.117.195:41451)
I20240502 01:58:14.875844 8559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21b7916993614f6d98c2f7460920a827" candidate_uuid: "7a9001351fe84a898b8544ef84aab5b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d87d866f3cbf4c14b4921a68f8779d54"
I20240502 01:58:14.875950 8559 raft_consensus.cc:3037] T 21b7916993614f6d98c2f7460920a827 P d87d866f3cbf4c14b4921a68f8779d54 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:14.875841 8484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21b7916993614f6d98c2f7460920a827" candidate_uuid: "7a9001351fe84a898b8544ef84aab5b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca98fb22a15140c6a26cca8192d0b6ba"
I20240502 01:58:14.876042 8484 raft_consensus.cc:3037] T 21b7916993614f6d98c2f7460920a827 P ca98fb22a15140c6a26cca8192d0b6ba [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:14.876713 8559 raft_consensus.cc:2454] T 21b7916993614f6d98c2f7460920a827 P d87d866f3cbf4c14b4921a68f8779d54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9001351fe84a898b8544ef84aab5b9 in term 1.
I20240502 01:58:14.876856 8484 raft_consensus.cc:2454] T 21b7916993614f6d98c2f7460920a827 P ca98fb22a15140c6a26cca8192d0b6ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9001351fe84a898b8544ef84aab5b9 in term 1.
I20240502 01:58:14.876987 8372 leader_election.cc:302] T 21b7916993614f6d98c2f7460920a827 P 7a9001351fe84a898b8544ef84aab5b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a9001351fe84a898b8544ef84aab5b9, d87d866f3cbf4c14b4921a68f8779d54; no voters:
I20240502 01:58:14.877125 8611 raft_consensus.cc:2781] T 21b7916993614f6d98c2f7460920a827 P 7a9001351fe84a898b8544ef84aab5b9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:14.877218 8611 raft_consensus.cc:686] T 21b7916993614f6d98c2f7460920a827 P 7a9001351fe84a898b8544ef84aab5b9 [term 1 LEADER]: Becoming Leader. State: Replica: 7a9001351fe84a898b8544ef84aab5b9, State: Running, Role: LEADER
I20240502 01:58:14.877357 8611 consensus_queue.cc:237] T 21b7916993614f6d98c2f7460920a827 P 7a9001351fe84a898b8544ef84aab5b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a9001351fe84a898b8544ef84aab5b9" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 43793 } } peers { permanent_uuid: "ca98fb22a15140c6a26cca8192d0b6ba" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 33585 } } peers { permanent_uuid: "d87d866f3cbf4c14b4921a68f8779d54" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 41451 } }
I20240502 01:58:14.878867 8304 catalog_manager.cc:5484] T 21b7916993614f6d98c2f7460920a827 P 7a9001351fe84a898b8544ef84aab5b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a9001351fe84a898b8544ef84aab5b9 (127.6.117.193). New cstate: current_term: 1 leader_uuid: "7a9001351fe84a898b8544ef84aab5b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a9001351fe84a898b8544ef84aab5b9" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 43793 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca98fb22a15140c6a26cca8192d0b6ba" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 33585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d87d866f3cbf4c14b4921a68f8779d54" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 41451 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:15.230053 8611 consensus_queue.cc:1035] T b404009b1e4e4be89b787476201c68a9 P 7a9001351fe84a898b8544ef84aab5b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca98fb22a15140c6a26cca8192d0b6ba" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 33585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:15.233160 8611 consensus_queue.cc:1035] T b404009b1e4e4be89b787476201c68a9 P 7a9001351fe84a898b8544ef84aab5b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d87d866f3cbf4c14b4921a68f8779d54" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:15.355542 8611 consensus_queue.cc:1035] T 21b7916993614f6d98c2f7460920a827 P 7a9001351fe84a898b8544ef84aab5b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca98fb22a15140c6a26cca8192d0b6ba" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 33585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:15.358588 8611 consensus_queue.cc:1035] T 21b7916993614f6d98c2f7460920a827 P 7a9001351fe84a898b8544ef84aab5b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d87d866f3cbf4c14b4921a68f8779d54" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:15.473788 8359 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:58:15.478055 8408 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b404009b1e4e4be89b787476201c68a9"
dest_uuid: "7a9001351fe84a898b8544ef84aab5b9"
mode: GRACEFUL
new_leader_uuid: "ca98fb22a15140c6a26cca8192d0b6ba"
from {username='slave'} at 127.0.0.1:59288
I20240502 01:58:15.478212 8408 raft_consensus.cc:595] T b404009b1e4e4be89b787476201c68a9 P 7a9001351fe84a898b8544ef84aab5b9 [term 1 LEADER]: Received request to transfer leadership to TS ca98fb22a15140c6a26cca8192d0b6ba
I20240502 01:58:15.478588 8359 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240502 01:58:15.478674 8359 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:58:15.788230 8635 raft_consensus.cc:979] T b404009b1e4e4be89b787476201c68a9 P 7a9001351fe84a898b8544ef84aab5b9: : Instructing follower ca98fb22a15140c6a26cca8192d0b6ba to start an election
I20240502 01:58:15.788388 8635 raft_consensus.cc:1067] T b404009b1e4e4be89b787476201c68a9 P 7a9001351fe84a898b8544ef84aab5b9 [term 1 LEADER]: Signalling peer ca98fb22a15140c6a26cca8192d0b6ba to start an election
I20240502 01:58:15.788815 8484 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b404009b1e4e4be89b787476201c68a9"
dest_uuid: "ca98fb22a15140c6a26cca8192d0b6ba"
from {username='slave'} at 127.0.0.1:42068
I20240502 01:58:15.788957 8484 raft_consensus.cc:482] T b404009b1e4e4be89b787476201c68a9 P ca98fb22a15140c6a26cca8192d0b6ba [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:58:15.789008 8484 raft_consensus.cc:3037] T b404009b1e4e4be89b787476201c68a9 P ca98fb22a15140c6a26cca8192d0b6ba [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:15.789794 8484 raft_consensus.cc:504] T b404009b1e4e4be89b787476201c68a9 P ca98fb22a15140c6a26cca8192d0b6ba [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca98fb22a15140c6a26cca8192d0b6ba" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 33585 } } peers { permanent_uuid: "7a9001351fe84a898b8544ef84aab5b9" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 43793 } } peers { permanent_uuid: "d87d866f3cbf4c14b4921a68f8779d54" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 41451 } }
I20240502 01:58:15.790388 8484 leader_election.cc:288] T b404009b1e4e4be89b787476201c68a9 P ca98fb22a15140c6a26cca8192d0b6ba [CANDIDATE]: Term 2 election: Requested vote from peers 7a9001351fe84a898b8544ef84aab5b9 (127.6.117.193:43793), d87d866f3cbf4c14b4921a68f8779d54 (127.6.117.195:41451)
I20240502 01:58:15.794798 8408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b404009b1e4e4be89b787476201c68a9" candidate_uuid: "ca98fb22a15140c6a26cca8192d0b6ba" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7a9001351fe84a898b8544ef84aab5b9"
I20240502 01:58:15.794799 8558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b404009b1e4e4be89b787476201c68a9" candidate_uuid: "ca98fb22a15140c6a26cca8192d0b6ba" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d87d866f3cbf4c14b4921a68f8779d54"
I20240502 01:58:15.794971 8408 raft_consensus.cc:3032] T b404009b1e4e4be89b787476201c68a9 P 7a9001351fe84a898b8544ef84aab5b9 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:58:15.794974 8558 raft_consensus.cc:3037] T b404009b1e4e4be89b787476201c68a9 P d87d866f3cbf4c14b4921a68f8779d54 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:15.795032 8408 raft_consensus.cc:729] T b404009b1e4e4be89b787476201c68a9 P 7a9001351fe84a898b8544ef84aab5b9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7a9001351fe84a898b8544ef84aab5b9, State: Running, Role: LEADER
I20240502 01:58:15.795217 8408 consensus_queue.cc:260] T b404009b1e4e4be89b787476201c68a9 P 7a9001351fe84a898b8544ef84aab5b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca98fb22a15140c6a26cca8192d0b6ba" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 33585 } } peers { permanent_uuid: "7a9001351fe84a898b8544ef84aab5b9" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 43793 } } peers { permanent_uuid: "d87d866f3cbf4c14b4921a68f8779d54" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 41451 } }
I20240502 01:58:15.795394 8408 raft_consensus.cc:3037] T b404009b1e4e4be89b787476201c68a9 P 7a9001351fe84a898b8544ef84aab5b9 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:15.795940 8558 raft_consensus.cc:2454] T b404009b1e4e4be89b787476201c68a9 P d87d866f3cbf4c14b4921a68f8779d54 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca98fb22a15140c6a26cca8192d0b6ba in term 2.
I20240502 01:58:15.796165 8408 raft_consensus.cc:2454] T b404009b1e4e4be89b787476201c68a9 P 7a9001351fe84a898b8544ef84aab5b9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca98fb22a15140c6a26cca8192d0b6ba in term 2.
I20240502 01:58:15.796228 8448 leader_election.cc:302] T b404009b1e4e4be89b787476201c68a9 P ca98fb22a15140c6a26cca8192d0b6ba [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ca98fb22a15140c6a26cca8192d0b6ba, d87d866f3cbf4c14b4921a68f8779d54; no voters:
I20240502 01:58:15.796453 8622 raft_consensus.cc:2781] T b404009b1e4e4be89b787476201c68a9 P ca98fb22a15140c6a26cca8192d0b6ba [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:58:15.796660 8622 raft_consensus.cc:686] T b404009b1e4e4be89b787476201c68a9 P ca98fb22a15140c6a26cca8192d0b6ba [term 2 LEADER]: Becoming Leader. State: Replica: ca98fb22a15140c6a26cca8192d0b6ba, State: Running, Role: LEADER
I20240502 01:58:15.796800 8622 consensus_queue.cc:237] T b404009b1e4e4be89b787476201c68a9 P ca98fb22a15140c6a26cca8192d0b6ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca98fb22a15140c6a26cca8192d0b6ba" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 33585 } } peers { permanent_uuid: "7a9001351fe84a898b8544ef84aab5b9" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 43793 } } peers { permanent_uuid: "d87d866f3cbf4c14b4921a68f8779d54" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 41451 } }
I20240502 01:58:15.798343 8304 catalog_manager.cc:5484] T b404009b1e4e4be89b787476201c68a9 P ca98fb22a15140c6a26cca8192d0b6ba reported cstate change: term changed from 1 to 2, leader changed from 7a9001351fe84a898b8544ef84aab5b9 (127.6.117.193) to ca98fb22a15140c6a26cca8192d0b6ba (127.6.117.194). New cstate: current_term: 2 leader_uuid: "ca98fb22a15140c6a26cca8192d0b6ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca98fb22a15140c6a26cca8192d0b6ba" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 33585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a9001351fe84a898b8544ef84aab5b9" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 43793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d87d866f3cbf4c14b4921a68f8779d54" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 41451 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:16.183137 8408 raft_consensus.cc:1261] T b404009b1e4e4be89b787476201c68a9 P 7a9001351fe84a898b8544ef84aab5b9 [term 2 FOLLOWER]: Refusing update from remote peer ca98fb22a15140c6a26cca8192d0b6ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 01:58:16.183607 8622 consensus_queue.cc:1035] T b404009b1e4e4be89b787476201c68a9 P ca98fb22a15140c6a26cca8192d0b6ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9001351fe84a898b8544ef84aab5b9" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 43793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:58:16.185870 8558 raft_consensus.cc:1261] T b404009b1e4e4be89b787476201c68a9 P d87d866f3cbf4c14b4921a68f8779d54 [term 2 FOLLOWER]: Refusing update from remote peer ca98fb22a15140c6a26cca8192d0b6ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 01:58:16.186338 8644 consensus_queue.cc:1035] T b404009b1e4e4be89b787476201c68a9 P ca98fb22a15140c6a26cca8192d0b6ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "d87d866f3cbf4c14b4921a68f8779d54" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:58:17.479004 8359 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:58:17.479398 8359 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:58:17.479460 8359 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:58:19.479771 8359 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:58:19.480124 8359 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:58:19.480182 8359 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:58:21.480479 8359 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:58:21.480821 8359 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:58:21.480878 8359 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:58:23.481163 8359 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:58:23.481544 8359 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:58:23.481622 8359 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:58:25.481937 8359 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:58:25.482395 8359 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:58:25.482465 8359 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:58:25.965346 6615 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 01:58:25.967929 6615 server_base.cc:1034] running on GCE node
W20240502 01:58:25.968348 8652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:25.968721 8654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:25.969029 8651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:58:25.969271 6615 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:58:25.969321 6615 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:58:25.969341 6615 hybrid_clock.cc:648] HybridClock initialized: now 1714615105969341 us; error 0 us; skew 500 ppm
I20240502 01:58:25.970431 6615 webserver.cc:460] Webserver started at http://127.6.117.196:39063/ using document root <none> and password file <none>
I20240502 01:58:25.970578 6615 fs_manager.cc:359] Metadata directory not provided
I20240502 01:58:25.970630 6615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:58:25.970707 6615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:58:25.971099 6615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714615076189642-6615-0/minicluster-data/ts-3-root/instance:
uuid: "2923209dd6094acc8d774378782f9ab7"
format_stamp: "Formatted at 2024-05-02 01:58:25 on dist-test-slave-89cm"
I20240502 01:58:25.972600 6615 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240502 01:58:25.973567 8659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:25.973821 6615 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240502 01:58:25.973896 6615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714615076189642-6615-0/minicluster-data/ts-3-root
uuid: "2923209dd6094acc8d774378782f9ab7"
format_stamp: "Formatted at 2024-05-02 01:58:25 on dist-test-slave-89cm"
I20240502 01:58:25.973960 6615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714615076189642-6615-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714615076189642-6615-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714615076189642-6615-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:58:25.981658 6615 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:58:25.982044 6615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:58:25.982353 6615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:58:25.982831 6615 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:58:25.982888 6615 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:25.982929 6615 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:58:25.982949 6615 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:25.987190 6615 rpc_server.cc:307] RPC server started. Bound to: 127.6.117.196:39711
I20240502 01:58:25.987232 8722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.117.196:39711 every 8 connection(s)
I20240502 01:58:25.991411 8723 heartbeater.cc:347] Connected to a master server at 127.6.117.254:36539
I20240502 01:58:25.991508 8723 heartbeater.cc:464] Registering TS with master...
I20240502 01:58:25.991714 8723 heartbeater.cc:511] Master 127.6.117.254:36539 requested a full tablet report, sending...
I20240502 01:58:25.992312 8304 ts_manager.cc:194] Registered new tserver with Master: 2923209dd6094acc8d774378782f9ab7 (127.6.117.196:39711)
I20240502 01:58:25.992854 8304 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38078
I20240502 01:58:26.488968 8484 consensus_queue.cc:237] T b404009b1e4e4be89b787476201c68a9 P ca98fb22a15140c6a26cca8192d0b6ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ca98fb22a15140c6a26cca8192d0b6ba" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 33585 } } peers { permanent_uuid: "7a9001351fe84a898b8544ef84aab5b9" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 43793 } attrs { replace: true } } peers { permanent_uuid: "d87d866f3cbf4c14b4921a68f8779d54" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 41451 } } peers { permanent_uuid: "2923209dd6094acc8d774378782f9ab7" member_type: NON_VOTER last_known_addr { host: "127.6.117.196" port: 39711 } attrs { promote: true } }
I20240502 01:58:26.490314 8408 raft_consensus.cc:1261] T b404009b1e4e4be89b787476201c68a9 P 7a9001351fe84a898b8544ef84aab5b9 [term 2 FOLLOWER]: Refusing update from remote peer ca98fb22a15140c6a26cca8192d0b6ba: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 01:58:26.490314 8558 raft_consensus.cc:1261] T b404009b1e4e4be89b787476201c68a9 P d87d866f3cbf4c14b4921a68f8779d54 [term 2 FOLLOWER]: Refusing update from remote peer ca98fb22a15140c6a26cca8192d0b6ba: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 01:58:26.490713 8730 consensus_queue.cc:1035] T b404009b1e4e4be89b787476201c68a9 P ca98fb22a15140c6a26cca8192d0b6ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "d87d866f3cbf4c14b4921a68f8779d54" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:58:26.490926 8649 consensus_queue.cc:1035] T b404009b1e4e4be89b787476201c68a9 P ca98fb22a15140c6a26cca8192d0b6ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9001351fe84a898b8544ef84aab5b9" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 43793 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:58:26.492550 8731 raft_consensus.cc:2932] T b404009b1e4e4be89b787476201c68a9 P ca98fb22a15140c6a26cca8192d0b6ba [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 2923209dd6094acc8d774378782f9ab7 (127.6.117.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ca98fb22a15140c6a26cca8192d0b6ba" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 33585 } } peers { permanent_uuid: "7a9001351fe84a898b8544ef84aab5b9" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 43793 } attrs { replace: true } } peers { permanent_uuid: "d87d866f3cbf4c14b4921a68f8779d54" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 41451 } } peers { permanent_uuid: "2923209dd6094acc8d774378782f9ab7" member_type: NON_VOTER last_known_addr { host: "127.6.117.196" port: 39711 } attrs { promote: true } } }
I20240502 01:58:26.492875 8408 raft_consensus.cc:2932] T b404009b1e4e4be89b787476201c68a9 P 7a9001351fe84a898b8544ef84aab5b9 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 2923209dd6094acc8d774378782f9ab7 (127.6.117.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ca98fb22a15140c6a26cca8192d0b6ba" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 33585 } } peers { permanent_uuid: "7a9001351fe84a898b8544ef84aab5b9" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 43793 } attrs { replace: true } } peers { permanent_uuid: "d87d866f3cbf4c14b4921a68f8779d54" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 41451 } } peers { permanent_uuid: "2923209dd6094acc8d774378782f9ab7" member_type: NON_VOTER last_known_addr { host: "127.6.117.196" port: 39711 } attrs { promote: true } } }
I20240502 01:58:26.493753 8558 raft_consensus.cc:2932] T b404009b1e4e4be89b787476201c68a9 P d87d866f3cbf4c14b4921a68f8779d54 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 2923209dd6094acc8d774378782f9ab7 (127.6.117.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ca98fb22a15140c6a26cca8192d0b6ba" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 33585 } } peers { permanent_uuid: "7a9001351fe84a898b8544ef84aab5b9" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 43793 } attrs { replace: true } } peers { permanent_uuid: "d87d866f3cbf4c14b4921a68f8779d54" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 41451 } } peers { permanent_uuid: "2923209dd6094acc8d774378782f9ab7" member_type: NON_VOTER last_known_addr { host: "127.6.117.196" port: 39711 } attrs { promote: true } } }
I20240502 01:58:26.495348 8304 catalog_manager.cc:5484] T b404009b1e4e4be89b787476201c68a9 P ca98fb22a15140c6a26cca8192d0b6ba reported cstate change: config changed from index -1 to 3, NON_VOTER 2923209dd6094acc8d774378782f9ab7 (127.6.117.196) added. New cstate: current_term: 2 leader_uuid: "ca98fb22a15140c6a26cca8192d0b6ba" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ca98fb22a15140c6a26cca8192d0b6ba" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 33585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a9001351fe84a898b8544ef84aab5b9" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 43793 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d87d866f3cbf4c14b4921a68f8779d54" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 41451 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2923209dd6094acc8d774378782f9ab7" member_type: NON_VOTER last_known_addr { host: "127.6.117.196" port: 39711 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 01:58:26.495630 8450 consensus_peers.cc:485] T b404009b1e4e4be89b787476201c68a9 P ca98fb22a15140c6a26cca8192d0b6ba -> Peer 2923209dd6094acc8d774378782f9ab7 (127.6.117.196:39711): Couldn't send request to peer 2923209dd6094acc8d774378782f9ab7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b404009b1e4e4be89b787476201c68a9. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:26.942624 8741 ts_tablet_manager.cc:902] T b404009b1e4e4be89b787476201c68a9 P 2923209dd6094acc8d774378782f9ab7: Initiating tablet copy from peer ca98fb22a15140c6a26cca8192d0b6ba (127.6.117.194:33585)
I20240502 01:58:26.943240 8741 tablet_copy_client.cc:308] T b404009b1e4e4be89b787476201c68a9 P 2923209dd6094acc8d774378782f9ab7: tablet copy: Beginning tablet copy session from remote peer at address 127.6.117.194:33585
I20240502 01:58:26.946579 8494 tablet_copy_service.cc:140] P ca98fb22a15140c6a26cca8192d0b6ba: Received BeginTabletCopySession request for tablet b404009b1e4e4be89b787476201c68a9 from peer 2923209dd6094acc8d774378782f9ab7 ({username='slave'} at 127.0.0.1:49502)
I20240502 01:58:26.946681 8494 tablet_copy_service.cc:161] P ca98fb22a15140c6a26cca8192d0b6ba: Beginning new tablet copy session on tablet b404009b1e4e4be89b787476201c68a9 from peer 2923209dd6094acc8d774378782f9ab7 at {username='slave'} at 127.0.0.1:49502: session id = 2923209dd6094acc8d774378782f9ab7-b404009b1e4e4be89b787476201c68a9
I20240502 01:58:26.947477 8494 tablet_copy_source_session.cc:205] T b404009b1e4e4be89b787476201c68a9 P ca98fb22a15140c6a26cca8192d0b6ba: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:26.948239 8741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b404009b1e4e4be89b787476201c68a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:26.950223 8741 tablet_copy_client.cc:783] T b404009b1e4e4be89b787476201c68a9 P 2923209dd6094acc8d774378782f9ab7: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:26.950400 8741 tablet_copy_client.cc:647] T b404009b1e4e4be89b787476201c68a9 P 2923209dd6094acc8d774378782f9ab7: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:26.951334 8741 tablet_copy_client.cc:523] T b404009b1e4e4be89b787476201c68a9 P 2923209dd6094acc8d774378782f9ab7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:26.952451 8741 tablet_bootstrap.cc:492] T b404009b1e4e4be89b787476201c68a9 P 2923209dd6094acc8d774378782f9ab7: Bootstrap starting.
I20240502 01:58:26.956300 8741 tablet_bootstrap.cc:492] T b404009b1e4e4be89b787476201c68a9 P 2923209dd6094acc8d774378782f9ab7: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:26.956535 8741 tablet_bootstrap.cc:492] T b404009b1e4e4be89b787476201c68a9 P 2923209dd6094acc8d774378782f9ab7: Bootstrap complete.
I20240502 01:58:26.956707 8741 ts_tablet_manager.cc:1367] T b404009b1e4e4be89b787476201c68a9 P 2923209dd6094acc8d774378782f9ab7: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.001s
I20240502 01:58:26.957156 8741 raft_consensus.cc:348] T b404009b1e4e4be89b787476201c68a9 P 2923209dd6094acc8d774378782f9ab7 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ca98fb22a15140c6a26cca8192d0b6ba" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 33585 } } peers { permanent_uuid: "7a9001351fe84a898b8544ef84aab5b9" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 43793 } attrs { replace: true } } peers { permanent_uuid: "d87d866f3cbf4c14b4921a68f8779d54" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 41451 } } peers { permanent_uuid: "2923209dd6094acc8d774378782f9ab7" member_type: NON_VOTER last_known_addr { host: "127.6.117.196" port: 39711 } attrs { promote: true } }
I20240502 01:58:26.957289 8741 raft_consensus.cc:729] T b404009b1e4e4be89b787476201c68a9 P 2923209dd6094acc8d774378782f9ab7 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 2923209dd6094acc8d774378782f9ab7, State: Initialized, Role: LEARNER
I20240502 01:58:26.957398 8741 consensus_queue.cc:260] T b404009b1e4e4be89b787476201c68a9 P 2923209dd6094acc8d774378782f9ab7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ca98fb22a15140c6a26cca8192d0b6ba" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 33585 } } peers { permanent_uuid: "7a9001351fe84a898b8544ef84aab5b9" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 43793 } attrs { replace: true } } peers { permanent_uuid: "d87d866f3cbf4c14b4921a68f8779d54" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 41451 } } peers { permanent_uuid: "2923209dd6094acc8d774378782f9ab7" member_type: NON_VOTER last_known_addr { host: "127.6.117.196" port: 39711 } attrs { promote: true } }
I20240502 01:58:26.957763 8741 ts_tablet_manager.cc:1398] T b404009b1e4e4be89b787476201c68a9 P 2923209dd6094acc8d774378782f9ab7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:26.957795 8723 heartbeater.cc:503] Master 127.6.117.254:36539 was elected leader, sending a full tablet report...
I20240502 01:58:26.958174 8494 tablet_copy_service.cc:342] P ca98fb22a15140c6a26cca8192d0b6ba: Request end of tablet copy session 2923209dd6094acc8d774378782f9ab7-b404009b1e4e4be89b787476201c68a9 received from {username='slave'} at 127.0.0.1:49502
I20240502 01:58:26.958254 8494 tablet_copy_service.cc:433] P ca98fb22a15140c6a26cca8192d0b6ba: ending tablet copy session 2923209dd6094acc8d774378782f9ab7-b404009b1e4e4be89b787476201c68a9 on tablet b404009b1e4e4be89b787476201c68a9 with peer 2923209dd6094acc8d774378782f9ab7
I20240502 01:58:27.368103 8697 raft_consensus.cc:1203] T b404009b1e4e4be89b787476201c68a9 P 2923209dd6094acc8d774378782f9ab7 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20240502 01:58:27.482679 8359 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:58:27.483031 8359 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:58:27.483098 8359 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:58:27.989431 8750 raft_consensus.cc:1050] T b404009b1e4e4be89b787476201c68a9 P ca98fb22a15140c6a26cca8192d0b6ba: attempting to promote NON_VOTER 2923209dd6094acc8d774378782f9ab7 to VOTER
I20240502 01:58:27.989930 8750 consensus_queue.cc:237] T b404009b1e4e4be89b787476201c68a9 P ca98fb22a15140c6a26cca8192d0b6ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca98fb22a15140c6a26cca8192d0b6ba" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 33585 } } peers { permanent_uuid: "7a9001351fe84a898b8544ef84aab5b9" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 43793 } attrs { replace: true } } peers { permanent_uuid: "d87d866f3cbf4c14b4921a68f8779d54" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 41451 } } peers { permanent_uuid: "2923209dd6094acc8d774378782f9ab7" member_type: VOTER last_known_addr { host: "127.6.117.196" port: 39711 } attrs { promote: false } }
I20240502 01:58:27.991111 8697 raft_consensus.cc:1261] T b404009b1e4e4be89b787476201c68a9 P 2923209dd6094acc8d774378782f9ab7 [term 2 LEARNER]: Refusing update from remote peer ca98fb22a15140c6a26cca8192d0b6ba: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:58:27.991290 8558 raft_consensus.cc:1261] T b404009b1e4e4be89b787476201c68a9 P d87d866f3cbf4c14b4921a68f8779d54 [term 2 FOLLOWER]: Refusing update from remote peer ca98fb22a15140c6a26cca8192d0b6ba: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:58:27.991284 8408 raft_consensus.cc:1261] T b404009b1e4e4be89b787476201c68a9 P 7a9001351fe84a898b8544ef84aab5b9 [term 2 FOLLOWER]: Refusing update from remote peer ca98fb22a15140c6a26cca8192d0b6ba: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:58:27.991523 8753 consensus_queue.cc:1035] T b404009b1e4e4be89b787476201c68a9 P ca98fb22a15140c6a26cca8192d0b6ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "2923209dd6094acc8d774378782f9ab7" member_type: VOTER last_known_addr { host: "127.6.117.196" port: 39711 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:27.991696 8751 consensus_queue.cc:1035] T b404009b1e4e4be89b787476201c68a9 P ca98fb22a15140c6a26cca8192d0b6ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "d87d866f3cbf4c14b4921a68f8779d54" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:27.991827 8750 consensus_queue.cc:1035] T b404009b1e4e4be89b787476201c68a9 P ca98fb22a15140c6a26cca8192d0b6ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9001351fe84a898b8544ef84aab5b9" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 43793 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:27.993489 8751 raft_consensus.cc:2932] T b404009b1e4e4be89b787476201c68a9 P ca98fb22a15140c6a26cca8192d0b6ba [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 2923209dd6094acc8d774378782f9ab7 (127.6.117.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca98fb22a15140c6a26cca8192d0b6ba" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 33585 } } peers { permanent_uuid: "7a9001351fe84a898b8544ef84aab5b9" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 43793 } attrs { replace: true } } peers { permanent_uuid: "d87d866f3cbf4c14b4921a68f8779d54" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 41451 } } peers { permanent_uuid: "2923209dd6094acc8d774378782f9ab7" member_type: VOTER last_known_addr { host: "127.6.117.196" port: 39711 } attrs { promote: false } } }
I20240502 01:58:27.993906 8558 raft_consensus.cc:2932] T b404009b1e4e4be89b787476201c68a9 P d87d866f3cbf4c14b4921a68f8779d54 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 2923209dd6094acc8d774378782f9ab7 (127.6.117.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca98fb22a15140c6a26cca8192d0b6ba" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 33585 } } peers { permanent_uuid: "7a9001351fe84a898b8544ef84aab5b9" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 43793 } attrs { replace: true } } peers { permanent_uuid: "d87d866f3cbf4c14b4921a68f8779d54" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 41451 } } peers { permanent_uuid: "2923209dd6094acc8d774378782f9ab7" member_type: VOTER last_known_addr { host: "127.6.117.196" port: 39711 } attrs { promote: false } } }
I20240502 01:58:27.994344 8408 raft_consensus.cc:2932] T b404009b1e4e4be89b787476201c68a9 P 7a9001351fe84a898b8544ef84aab5b9 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 2923209dd6094acc8d774378782f9ab7 (127.6.117.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca98fb22a15140c6a26cca8192d0b6ba" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 33585 } } peers { permanent_uuid: "7a9001351fe84a898b8544ef84aab5b9" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 43793 } attrs { replace: true } } peers { permanent_uuid: "d87d866f3cbf4c14b4921a68f8779d54" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 41451 } } peers { permanent_uuid: "2923209dd6094acc8d774378782f9ab7" member_type: VOTER last_known_addr { host: "127.6.117.196" port: 39711 } attrs { promote: false } } }
I20240502 01:58:27.995766 8697 raft_consensus.cc:2932] T b404009b1e4e4be89b787476201c68a9 P 2923209dd6094acc8d774378782f9ab7 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 2923209dd6094acc8d774378782f9ab7 (127.6.117.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca98fb22a15140c6a26cca8192d0b6ba" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 33585 } } peers { permanent_uuid: "7a9001351fe84a898b8544ef84aab5b9" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 43793 } attrs { replace: true } } peers { permanent_uuid: "d87d866f3cbf4c14b4921a68f8779d54" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 41451 } } peers { permanent_uuid: "2923209dd6094acc8d774378782f9ab7" member_type: VOTER last_known_addr { host: "127.6.117.196" port: 39711 } attrs { promote: false } } }
I20240502 01:58:27.996703 8303 catalog_manager.cc:5484] T b404009b1e4e4be89b787476201c68a9 P ca98fb22a15140c6a26cca8192d0b6ba reported cstate change: config changed from index 3 to 4, 2923209dd6094acc8d774378782f9ab7 (127.6.117.196) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "ca98fb22a15140c6a26cca8192d0b6ba" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca98fb22a15140c6a26cca8192d0b6ba" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 33585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a9001351fe84a898b8544ef84aab5b9" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 43793 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d87d866f3cbf4c14b4921a68f8779d54" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 41451 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2923209dd6094acc8d774378782f9ab7" member_type: VOTER last_known_addr { host: "127.6.117.196" port: 39711 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:28.000193 8483 consensus_queue.cc:237] T b404009b1e4e4be89b787476201c68a9 P ca98fb22a15140c6a26cca8192d0b6ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ca98fb22a15140c6a26cca8192d0b6ba" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 33585 } } peers { permanent_uuid: "d87d866f3cbf4c14b4921a68f8779d54" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 41451 } } peers { permanent_uuid: "2923209dd6094acc8d774378782f9ab7" member_type: VOTER last_known_addr { host: "127.6.117.196" port: 39711 } attrs { promote: false } }
I20240502 01:58:28.001170 8697 raft_consensus.cc:1261] T b404009b1e4e4be89b787476201c68a9 P 2923209dd6094acc8d774378782f9ab7 [term 2 FOLLOWER]: Refusing update from remote peer ca98fb22a15140c6a26cca8192d0b6ba: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 01:58:28.001446 8558 raft_consensus.cc:1261] T b404009b1e4e4be89b787476201c68a9 P d87d866f3cbf4c14b4921a68f8779d54 [term 2 FOLLOWER]: Refusing update from remote peer ca98fb22a15140c6a26cca8192d0b6ba: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 01:58:28.001454 8753 consensus_queue.cc:1035] T b404009b1e4e4be89b787476201c68a9 P ca98fb22a15140c6a26cca8192d0b6ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "2923209dd6094acc8d774378782f9ab7" member_type: VOTER last_known_addr { host: "127.6.117.196" port: 39711 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:28.001739 8750 consensus_queue.cc:1035] T b404009b1e4e4be89b787476201c68a9 P ca98fb22a15140c6a26cca8192d0b6ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "d87d866f3cbf4c14b4921a68f8779d54" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:28.002769 8753 raft_consensus.cc:2932] T b404009b1e4e4be89b787476201c68a9 P ca98fb22a15140c6a26cca8192d0b6ba [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 7a9001351fe84a898b8544ef84aab5b9 (127.6.117.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ca98fb22a15140c6a26cca8192d0b6ba" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 33585 } } peers { permanent_uuid: "d87d866f3cbf4c14b4921a68f8779d54" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 41451 } } peers { permanent_uuid: "2923209dd6094acc8d774378782f9ab7" member_type: VOTER last_known_addr { host: "127.6.117.196" port: 39711 } attrs { promote: false } } }
I20240502 01:58:28.003456 8558 raft_consensus.cc:2932] T b404009b1e4e4be89b787476201c68a9 P d87d866f3cbf4c14b4921a68f8779d54 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 7a9001351fe84a898b8544ef84aab5b9 (127.6.117.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ca98fb22a15140c6a26cca8192d0b6ba" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 33585 } } peers { permanent_uuid: "d87d866f3cbf4c14b4921a68f8779d54" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 41451 } } peers { permanent_uuid: "2923209dd6094acc8d774378782f9ab7" member_type: VOTER last_known_addr { host: "127.6.117.196" port: 39711 } attrs { promote: false } } }
I20240502 01:58:28.004159 8291 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b404009b1e4e4be89b787476201c68a9 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:58:28.004386 8697 raft_consensus.cc:2932] T b404009b1e4e4be89b787476201c68a9 P 2923209dd6094acc8d774378782f9ab7 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 7a9001351fe84a898b8544ef84aab5b9 (127.6.117.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ca98fb22a15140c6a26cca8192d0b6ba" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 33585 } } peers { permanent_uuid: "d87d866f3cbf4c14b4921a68f8779d54" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 41451 } } peers { permanent_uuid: "2923209dd6094acc8d774378782f9ab7" member_type: VOTER last_known_addr { host: "127.6.117.196" port: 39711 } attrs { promote: false } } }
I20240502 01:58:28.004741 8304 catalog_manager.cc:5484] T b404009b1e4e4be89b787476201c68a9 P ca98fb22a15140c6a26cca8192d0b6ba reported cstate change: config changed from index 4 to 5, VOTER 7a9001351fe84a898b8544ef84aab5b9 (127.6.117.193) evicted. New cstate: current_term: 2 leader_uuid: "ca98fb22a15140c6a26cca8192d0b6ba" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ca98fb22a15140c6a26cca8192d0b6ba" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 33585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d87d866f3cbf4c14b4921a68f8779d54" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 41451 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2923209dd6094acc8d774378782f9ab7" member_type: VOTER last_known_addr { host: "127.6.117.196" port: 39711 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:28.007613 8398 tablet_service.cc:1514] Processing DeleteTablet for tablet b404009b1e4e4be89b787476201c68a9 with delete_type TABLET_DATA_TOMBSTONED (TS 7a9001351fe84a898b8544ef84aab5b9 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:59278
I20240502 01:58:28.008138 8764 tablet_replica.cc:330] T b404009b1e4e4be89b787476201c68a9 P 7a9001351fe84a898b8544ef84aab5b9: stopping tablet replica
I20240502 01:58:28.008306 8764 raft_consensus.cc:2229] T b404009b1e4e4be89b787476201c68a9 P 7a9001351fe84a898b8544ef84aab5b9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:28.008572 8764 raft_consensus.cc:2258] T b404009b1e4e4be89b787476201c68a9 P 7a9001351fe84a898b8544ef84aab5b9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:28.009148 8764 ts_tablet_manager.cc:1875] T b404009b1e4e4be89b787476201c68a9 P 7a9001351fe84a898b8544ef84aab5b9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:28.010784 8764 ts_tablet_manager.cc:1888] T b404009b1e4e4be89b787476201c68a9 P 7a9001351fe84a898b8544ef84aab5b9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240502 01:58:28.010851 8764 log.cc:1196] T b404009b1e4e4be89b787476201c68a9 P 7a9001351fe84a898b8544ef84aab5b9: Deleting WAL directory at /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714615076189642-6615-0/minicluster-data/ts-0-root/wals/b404009b1e4e4be89b787476201c68a9
I20240502 01:58:28.011276 8291 catalog_manager.cc:4830] TS 7a9001351fe84a898b8544ef84aab5b9 (127.6.117.193:43793): tablet b404009b1e4e4be89b787476201c68a9 (table test-workload [id=a959bafba1434effa4b51918619ee5c1]) successfully deleted
I20240502 01:58:29.483433 8359 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:58:29.483839 8359 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:58:29.483911 8359 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:58:31.484349 8359 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:58:31.484762 8359 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:58:31.484836 8359 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:58:33.485186 8359 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:58:33.485612 8359 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:58:33.485687 8359 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:58:35.486023 8359 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:58:35.486464 8359 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:58:35.486537 8359 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:58:37.486894 8359 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:58:37.487346 8359 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:58:37.487413 8359 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:58:39.487766 8359 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:58:39.488186 8359 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:58:39.488281 8359 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:58:40.065384 6615 tablet_server.cc:178] TabletServer@127.6.117.193:0 shutting down...
I20240502 01:58:40.068892 6615 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:58:40.069105 6615 tablet_replica.cc:330] T 21b7916993614f6d98c2f7460920a827 P 7a9001351fe84a898b8544ef84aab5b9: stopping tablet replica
I20240502 01:58:40.069221 6615 raft_consensus.cc:2229] T 21b7916993614f6d98c2f7460920a827 P 7a9001351fe84a898b8544ef84aab5b9 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:58:40.069442 6615 raft_consensus.cc:2258] T 21b7916993614f6d98c2f7460920a827 P 7a9001351fe84a898b8544ef84aab5b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:40.082477 6615 tablet_server.cc:194] TabletServer@127.6.117.193:0 shutdown complete.
I20240502 01:58:40.084795 6615 tablet_server.cc:178] TabletServer@127.6.117.194:0 shutting down...
I20240502 01:58:40.088655 6615 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:58:40.088842 6615 tablet_replica.cc:330] T 21b7916993614f6d98c2f7460920a827 P ca98fb22a15140c6a26cca8192d0b6ba: stopping tablet replica
I20240502 01:58:40.088963 6615 raft_consensus.cc:2229] T 21b7916993614f6d98c2f7460920a827 P ca98fb22a15140c6a26cca8192d0b6ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:40.089078 6615 raft_consensus.cc:2258] T 21b7916993614f6d98c2f7460920a827 P ca98fb22a15140c6a26cca8192d0b6ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:40.089493 6615 tablet_replica.cc:330] T b404009b1e4e4be89b787476201c68a9 P ca98fb22a15140c6a26cca8192d0b6ba: stopping tablet replica
I20240502 01:58:40.089566 6615 raft_consensus.cc:2229] T b404009b1e4e4be89b787476201c68a9 P ca98fb22a15140c6a26cca8192d0b6ba [term 2 LEADER]: Raft consensus shutting down.
I20240502 01:58:40.089720 6615 raft_consensus.cc:2258] T b404009b1e4e4be89b787476201c68a9 P ca98fb22a15140c6a26cca8192d0b6ba [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:40.103477 6615 tablet_server.cc:194] TabletServer@127.6.117.194:0 shutdown complete.
I20240502 01:58:40.106098 6615 tablet_server.cc:178] TabletServer@127.6.117.195:0 shutting down...
I20240502 01:58:40.110196 6615 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:58:40.110391 6615 tablet_replica.cc:330] T 21b7916993614f6d98c2f7460920a827 P d87d866f3cbf4c14b4921a68f8779d54: stopping tablet replica
I20240502 01:58:40.110517 6615 raft_consensus.cc:2229] T 21b7916993614f6d98c2f7460920a827 P d87d866f3cbf4c14b4921a68f8779d54 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:40.110642 6615 raft_consensus.cc:2258] T 21b7916993614f6d98c2f7460920a827 P d87d866f3cbf4c14b4921a68f8779d54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:40.111079 6615 tablet_replica.cc:330] T b404009b1e4e4be89b787476201c68a9 P d87d866f3cbf4c14b4921a68f8779d54: stopping tablet replica
I20240502 01:58:40.111166 6615 raft_consensus.cc:2229] T b404009b1e4e4be89b787476201c68a9 P d87d866f3cbf4c14b4921a68f8779d54 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:40.111292 6615 raft_consensus.cc:2258] T b404009b1e4e4be89b787476201c68a9 P d87d866f3cbf4c14b4921a68f8779d54 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:40.124909 6615 tablet_server.cc:194] TabletServer@127.6.117.195:0 shutdown complete.
I20240502 01:58:40.127142 6615 tablet_server.cc:178] TabletServer@127.6.117.196:0 shutting down...
I20240502 01:58:40.130965 6615 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:58:40.131137 6615 tablet_replica.cc:330] T b404009b1e4e4be89b787476201c68a9 P 2923209dd6094acc8d774378782f9ab7: stopping tablet replica
I20240502 01:58:40.131242 6615 raft_consensus.cc:2229] T b404009b1e4e4be89b787476201c68a9 P 2923209dd6094acc8d774378782f9ab7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:40.131376 6615 raft_consensus.cc:2258] T b404009b1e4e4be89b787476201c68a9 P 2923209dd6094acc8d774378782f9ab7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:40.143813 6615 tablet_server.cc:194] TabletServer@127.6.117.196:0 shutdown complete.
I20240502 01:58:40.145921 6615 master.cc:537] Master@127.6.117.254:36539 shutting down...
I20240502 01:58:40.149988 6615 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 6ac6781202f74b91be784b5f5196b90f [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:58:40.150152 6615 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 6ac6781202f74b91be784b5f5196b90f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:40.150221 6615 tablet_replica.cc:330] T 00000000000000000000000000000000 P 6ac6781202f74b91be784b5f5196b90f: stopping tablet replica
I20240502 01:58:40.162707 6615 master.cc:552] Master@127.6.117.254:36539 shutdown complete.
[ OK ] AutoRebalancerTest.MovesScheduledIfAddTserver (26732 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfNoTservers
I20240502 01:58:40.169262 6615 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.6.117.254:42669
I20240502 01:58:40.169576 6615 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:58:40.171732 8767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:58:40.172201 6615 server_base.cc:1034] running on GCE node
W20240502 01:58:40.172606 8770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:40.172979 8768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:58:40.173149 6615 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:58:40.173189 6615 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:58:40.173203 6615 hybrid_clock.cc:648] HybridClock initialized: now 1714615120173203 us; error 0 us; skew 500 ppm
I20240502 01:58:40.179594 6615 webserver.cc:460] Webserver started at http://127.6.117.254:35753/ using document root <none> and password file <none>
I20240502 01:58:40.179746 6615 fs_manager.cc:359] Metadata directory not provided
I20240502 01:58:40.179787 6615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:58:40.179847 6615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:58:40.180199 6615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714615076189642-6615-0/minicluster-data/master-0-root/instance:
uuid: "b9f2814681ee4a58a7ceb59d7350a9b4"
format_stamp: "Formatted at 2024-05-02 01:58:40 on dist-test-slave-89cm"
I20240502 01:58:40.181640 6615 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240502 01:58:40.182471 8775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:40.182729 6615 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240502 01:58:40.182798 6615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714615076189642-6615-0/minicluster-data/master-0-root
uuid: "b9f2814681ee4a58a7ceb59d7350a9b4"
format_stamp: "Formatted at 2024-05-02 01:58:40 on dist-test-slave-89cm"
I20240502 01:58:40.182866 6615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714615076189642-6615-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714615076189642-6615-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714615076189642-6615-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:58:40.190500 6615 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:58:40.190843 6615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:58:40.194630 6615 rpc_server.cc:307] RPC server started. Bound to: 127.6.117.254:42669
I20240502 01:58:40.196703 8827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.117.254:42669 every 8 connection(s)
I20240502 01:58:40.197139 8828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:40.198892 8828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9f2814681ee4a58a7ceb59d7350a9b4: Bootstrap starting.
I20240502 01:58:40.199630 8828 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9f2814681ee4a58a7ceb59d7350a9b4: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:40.200614 8828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9f2814681ee4a58a7ceb59d7350a9b4: No bootstrap required, opened a new log
I20240502 01:58:40.200978 8828 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9f2814681ee4a58a7ceb59d7350a9b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9f2814681ee4a58a7ceb59d7350a9b4" member_type: VOTER }
I20240502 01:58:40.201073 8828 raft_consensus.cc:374] T 00000000000000000000000000000000 P b9f2814681ee4a58a7ceb59d7350a9b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:40.201103 8828 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9f2814681ee4a58a7ceb59d7350a9b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9f2814681ee4a58a7ceb59d7350a9b4, State: Initialized, Role: FOLLOWER
I20240502 01:58:40.201228 8828 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9f2814681ee4a58a7ceb59d7350a9b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9f2814681ee4a58a7ceb59d7350a9b4" member_type: VOTER }
I20240502 01:58:40.201297 8828 raft_consensus.cc:388] T 00000000000000000000000000000000 P b9f2814681ee4a58a7ceb59d7350a9b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:58:40.201333 8828 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9f2814681ee4a58a7ceb59d7350a9b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:58:40.201381 8828 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9f2814681ee4a58a7ceb59d7350a9b4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:40.202023 8828 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9f2814681ee4a58a7ceb59d7350a9b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9f2814681ee4a58a7ceb59d7350a9b4" member_type: VOTER }
I20240502 01:58:40.202150 8828 leader_election.cc:302] T 00000000000000000000000000000000 P b9f2814681ee4a58a7ceb59d7350a9b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9f2814681ee4a58a7ceb59d7350a9b4; no voters:
I20240502 01:58:40.202327 8828 leader_election.cc:288] T 00000000000000000000000000000000 P b9f2814681ee4a58a7ceb59d7350a9b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:58:40.202471 8831 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9f2814681ee4a58a7ceb59d7350a9b4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:40.202672 8831 raft_consensus.cc:686] T 00000000000000000000000000000000 P b9f2814681ee4a58a7ceb59d7350a9b4 [term 1 LEADER]: Becoming Leader. State: Replica: b9f2814681ee4a58a7ceb59d7350a9b4, State: Running, Role: LEADER
I20240502 01:58:40.202746 8828 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9f2814681ee4a58a7ceb59d7350a9b4 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 01:58:40.202807 8831 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9f2814681ee4a58a7ceb59d7350a9b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9f2814681ee4a58a7ceb59d7350a9b4" member_type: VOTER }
I20240502 01:58:40.203207 8832 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9f2814681ee4a58a7ceb59d7350a9b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b9f2814681ee4a58a7ceb59d7350a9b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9f2814681ee4a58a7ceb59d7350a9b4" member_type: VOTER } }
I20240502 01:58:40.203248 8833 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9f2814681ee4a58a7ceb59d7350a9b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9f2814681ee4a58a7ceb59d7350a9b4. Latest consensus state: current_term: 1 leader_uuid: "b9f2814681ee4a58a7ceb59d7350a9b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9f2814681ee4a58a7ceb59d7350a9b4" member_type: VOTER } }
I20240502 01:58:40.203295 8832 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9f2814681ee4a58a7ceb59d7350a9b4 [sys.catalog]: This master's current role is: LEADER
I20240502 01:58:40.203339 8833 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9f2814681ee4a58a7ceb59d7350a9b4 [sys.catalog]: This master's current role is: LEADER
I20240502 01:58:40.203819 8837 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 01:58:40.204578 8837 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 01:58:40.204818 6615 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 01:58:40.206350 8837 catalog_manager.cc:1347] Generated new cluster ID: 30b9a34b462b448abd2476b31048a077
I20240502 01:58:40.206393 8837 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 01:58:40.213891 8837 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 01:58:40.214421 8837 catalog_manager.cc:1504] Loading token signing keys...
I20240502 01:58:40.219837 8837 catalog_manager.cc:5857] T 00000000000000000000000000000000 P b9f2814681ee4a58a7ceb59d7350a9b4: Generated new TSK 0
I20240502 01:58:40.220010 8837 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 01:58:40.236755 6615 internal_mini_cluster.cc:371] 0 TS(s) registered with all masters after 7.9477e-05s
I20240502 01:58:52.315156 6615 master.cc:537] Master@127.6.117.254:42669 shutting down...
I20240502 01:58:52.318081 6615 raft_consensus.cc:2229] T 00000000000000000000000000000000 P b9f2814681ee4a58a7ceb59d7350a9b4 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:58:52.318259 6615 raft_consensus.cc:2258] T 00000000000000000000000000000000 P b9f2814681ee4a58a7ceb59d7350a9b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:52.318308 6615 tablet_replica.cc:330] T 00000000000000000000000000000000 P b9f2814681ee4a58a7ceb59d7350a9b4: stopping tablet replica
I20240502 01:58:52.330430 6615 master.cc:552] Master@127.6.117.254:42669 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfNoTservers (12165 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfNoTablets
I20240502 01:58:52.334484 6615 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.6.117.254:43557
I20240502 01:58:52.334801 6615 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:58:52.336773 8853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:52.337265 8850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:52.337622 8851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:58:52.338052 6615 server_base.cc:1034] running on GCE node
I20240502 01:58:52.338186 6615 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:58:52.338222 6615 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:58:52.338241 6615 hybrid_clock.cc:648] HybridClock initialized: now 1714615132338241 us; error 0 us; skew 500 ppm
I20240502 01:58:52.344661 6615 webserver.cc:460] Webserver started at http://127.6.117.254:39347/ using document root <none> and password file <none>
I20240502 01:58:52.344844 6615 fs_manager.cc:359] Metadata directory not provided
I20240502 01:58:52.344900 6615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:58:52.344982 6615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:58:52.345417 6615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714615076189642-6615-0/minicluster-data/master-0-root/instance:
uuid: "1433c29a836c48d8934cab39b6a12ce5"
format_stamp: "Formatted at 2024-05-02 01:58:52 on dist-test-slave-89cm"
I20240502 01:58:52.346971 6615 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240502 01:58:52.347970 8858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:52.348325 6615 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240502 01:58:52.348414 6615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714615076189642-6615-0/minicluster-data/master-0-root
uuid: "1433c29a836c48d8934cab39b6a12ce5"
format_stamp: "Formatted at 2024-05-02 01:58:52 on dist-test-slave-89cm"
I20240502 01:58:52.348490 6615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714615076189642-6615-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714615076189642-6615-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714615076189642-6615-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:58:52.357300 6615 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:58:52.357671 6615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:58:52.361409 6615 rpc_server.cc:307] RPC server started. Bound to: 127.6.117.254:43557
I20240502 01:58:52.361977 8910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.117.254:43557 every 8 connection(s)
I20240502 01:58:52.362423 8911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:52.364171 8911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1433c29a836c48d8934cab39b6a12ce5: Bootstrap starting.
I20240502 01:58:52.364926 8911 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1433c29a836c48d8934cab39b6a12ce5: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:52.365868 8911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1433c29a836c48d8934cab39b6a12ce5: No bootstrap required, opened a new log
I20240502 01:58:52.366223 8911 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1433c29a836c48d8934cab39b6a12ce5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1433c29a836c48d8934cab39b6a12ce5" member_type: VOTER }
I20240502 01:58:52.366312 8911 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1433c29a836c48d8934cab39b6a12ce5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:52.366344 8911 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1433c29a836c48d8934cab39b6a12ce5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1433c29a836c48d8934cab39b6a12ce5, State: Initialized, Role: FOLLOWER
I20240502 01:58:52.366485 8911 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1433c29a836c48d8934cab39b6a12ce5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1433c29a836c48d8934cab39b6a12ce5" member_type: VOTER }
I20240502 01:58:52.366555 8911 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1433c29a836c48d8934cab39b6a12ce5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:58:52.366587 8911 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1433c29a836c48d8934cab39b6a12ce5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:58:52.366631 8911 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1433c29a836c48d8934cab39b6a12ce5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:52.367254 8911 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1433c29a836c48d8934cab39b6a12ce5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1433c29a836c48d8934cab39b6a12ce5" member_type: VOTER }
I20240502 01:58:52.367368 8911 leader_election.cc:302] T 00000000000000000000000000000000 P 1433c29a836c48d8934cab39b6a12ce5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1433c29a836c48d8934cab39b6a12ce5; no voters:
I20240502 01:58:52.367544 8911 leader_election.cc:288] T 00000000000000000000000000000000 P 1433c29a836c48d8934cab39b6a12ce5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:58:52.367631 8914 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1433c29a836c48d8934cab39b6a12ce5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:52.367821 8914 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1433c29a836c48d8934cab39b6a12ce5 [term 1 LEADER]: Becoming Leader. State: Replica: 1433c29a836c48d8934cab39b6a12ce5, State: Running, Role: LEADER
I20240502 01:58:52.367954 8911 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1433c29a836c48d8934cab39b6a12ce5 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 01:58:52.367944 8914 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1433c29a836c48d8934cab39b6a12ce5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1433c29a836c48d8934cab39b6a12ce5" member_type: VOTER }
I20240502 01:58:52.368372 8915 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1433c29a836c48d8934cab39b6a12ce5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1433c29a836c48d8934cab39b6a12ce5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1433c29a836c48d8934cab39b6a12ce5" member_type: VOTER } }
I20240502 01:58:52.368395 8916 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1433c29a836c48d8934cab39b6a12ce5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1433c29a836c48d8934cab39b6a12ce5. Latest consensus state: current_term: 1 leader_uuid: "1433c29a836c48d8934cab39b6a12ce5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1433c29a836c48d8934cab39b6a12ce5" member_type: VOTER } }
I20240502 01:58:52.368463 8915 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1433c29a836c48d8934cab39b6a12ce5 [sys.catalog]: This master's current role is: LEADER
I20240502 01:58:52.368487 8916 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1433c29a836c48d8934cab39b6a12ce5 [sys.catalog]: This master's current role is: LEADER
I20240502 01:58:52.368726 8918 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 01:58:52.369709 8918 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 01:58:52.369951 6615 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 01:58:52.371431 8918 catalog_manager.cc:1347] Generated new cluster ID: 660c24768cfd47a98483a75a2f0f2226
I20240502 01:58:52.371482 8918 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 01:58:52.382122 8918 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 01:58:52.382692 8918 catalog_manager.cc:1504] Loading token signing keys...
I20240502 01:58:52.387701 8918 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 1433c29a836c48d8934cab39b6a12ce5: Generated new TSK 0
I20240502 01:58:52.387862 8918 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 01:58:52.402303 6615 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:58:52.404583 8932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:52.404934 8935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:52.405361 8933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:58:52.405778 6615 server_base.cc:1034] running on GCE node
I20240502 01:58:52.405930 6615 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:58:52.405967 6615 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:58:52.405987 6615 hybrid_clock.cc:648] HybridClock initialized: now 1714615132405987 us; error 0 us; skew 500 ppm
I20240502 01:58:52.406885 6615 webserver.cc:460] Webserver started at http://127.6.117.193:45887/ using document root <none> and password file <none>
I20240502 01:58:52.407032 6615 fs_manager.cc:359] Metadata directory not provided
I20240502 01:58:52.407084 6615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:58:52.407159 6615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:58:52.407558 6615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714615076189642-6615-0/minicluster-data/ts-0-root/instance:
uuid: "d47808e3d72442ab8ea2723d4fa5d1d6"
format_stamp: "Formatted at 2024-05-02 01:58:52 on dist-test-slave-89cm"
I20240502 01:58:52.408974 6615 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240502 01:58:52.409873 8940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:52.410095 6615 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240502 01:58:52.410158 6615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714615076189642-6615-0/minicluster-data/ts-0-root
uuid: "d47808e3d72442ab8ea2723d4fa5d1d6"
format_stamp: "Formatted at 2024-05-02 01:58:52 on dist-test-slave-89cm"
I20240502 01:58:52.410224 6615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714615076189642-6615-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714615076189642-6615-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714615076189642-6615-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:58:52.436584 6615 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:58:52.436964 6615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:58:52.437258 6615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:58:52.437736 6615 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:58:52.437776 6615 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:52.437816 6615 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:58:52.437839 6615 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:52.441685 6615 rpc_server.cc:307] RPC server started. Bound to: 127.6.117.193:36445
I20240502 01:58:52.441720 9003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.117.193:36445 every 8 connection(s)
I20240502 01:58:52.442445 6615 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:58:52.444449 9008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:52.444869 9011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:58:52.445420 6615 server_base.cc:1034] running on GCE node
W20240502 01:58:52.445593 9009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:58:52.445824 6615 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:58:52.445868 6615 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:58:52.445881 6615 hybrid_clock.cc:648] HybridClock initialized: now 1714615132445881 us; error 0 us; skew 500 ppm
I20240502 01:58:52.446641 9004 heartbeater.cc:347] Connected to a master server at 127.6.117.254:43557
I20240502 01:58:52.446712 6615 webserver.cc:460] Webserver started at http://127.6.117.194:33813/ using document root <none> and password file <none>
I20240502 01:58:52.446734 9004 heartbeater.cc:464] Registering TS with master...
I20240502 01:58:52.446868 6615 fs_manager.cc:359] Metadata directory not provided
I20240502 01:58:52.446915 6615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:58:52.446961 9004 heartbeater.cc:511] Master 127.6.117.254:43557 requested a full tablet report, sending...
I20240502 01:58:52.446985 6615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:58:52.447420 6615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714615076189642-6615-0/minicluster-data/ts-1-root/instance:
uuid: "46bcf62d01bd434e8e7a7b6356ea165d"
format_stamp: "Formatted at 2024-05-02 01:58:52 on dist-test-slave-89cm"
I20240502 01:58:52.447631 8875 ts_manager.cc:194] Registered new tserver with Master: d47808e3d72442ab8ea2723d4fa5d1d6 (127.6.117.193:36445)
I20240502 01:58:52.448381 8875 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47084
I20240502 01:58:52.448945 6615 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240502 01:58:52.449889 9016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:52.450155 6615 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240502 01:58:52.450213 6615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714615076189642-6615-0/minicluster-data/ts-1-root
uuid: "46bcf62d01bd434e8e7a7b6356ea165d"
format_stamp: "Formatted at 2024-05-02 01:58:52 on dist-test-slave-89cm"
I20240502 01:58:52.450269 6615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714615076189642-6615-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714615076189642-6615-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714615076189642-6615-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:58:52.464524 6615 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:58:52.464916 6615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:58:52.465164 6615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:58:52.465627 6615 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:58:52.465667 6615 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:52.465698 6615 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:58:52.465714 6615 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:52.469677 6615 rpc_server.cc:307] RPC server started. Bound to: 127.6.117.194:40275
I20240502 01:58:52.469751 9079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.117.194:40275 every 8 connection(s)
I20240502 01:58:52.470410 6615 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:58:52.472592 9083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:52.473014 9084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:52.473378 9086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:58:52.473814 6615 server_base.cc:1034] running on GCE node
I20240502 01:58:52.474022 6615 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:58:52.474076 6615 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:58:52.474095 6615 hybrid_clock.cc:648] HybridClock initialized: now 1714615132474095 us; error 0 us; skew 500 ppm
I20240502 01:58:52.474408 9080 heartbeater.cc:347] Connected to a master server at 127.6.117.254:43557
I20240502 01:58:52.474517 9080 heartbeater.cc:464] Registering TS with master...
I20240502 01:58:52.474751 9080 heartbeater.cc:511] Master 127.6.117.254:43557 requested a full tablet report, sending...
I20240502 01:58:52.475004 6615 webserver.cc:460] Webserver started at http://127.6.117.195:34443/ using document root <none> and password file <none>
I20240502 01:58:52.475159 6615 fs_manager.cc:359] Metadata directory not provided
I20240502 01:58:52.475212 6615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:58:52.475276 6615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:58:52.475351 8875 ts_manager.cc:194] Registered new tserver with Master: 46bcf62d01bd434e8e7a7b6356ea165d (127.6.117.194:40275)
I20240502 01:58:52.475668 6615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714615076189642-6615-0/minicluster-data/ts-2-root/instance:
uuid: "5afe38d1059d4f6f80b1aef4810efaa8"
format_stamp: "Formatted at 2024-05-02 01:58:52 on dist-test-slave-89cm"
I20240502 01:58:52.475884 8875 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47090
I20240502 01:58:52.477124 6615 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240502 01:58:52.477963 9091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:52.478209 6615 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:52.478273 6615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714615076189642-6615-0/minicluster-data/ts-2-root
uuid: "5afe38d1059d4f6f80b1aef4810efaa8"
format_stamp: "Formatted at 2024-05-02 01:58:52 on dist-test-slave-89cm"
I20240502 01:58:52.478339 6615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714615076189642-6615-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714615076189642-6615-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714615076189642-6615-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:58:52.490940 6615 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:58:52.491322 6615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:58:52.491604 6615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:58:52.492051 6615 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:58:52.492091 6615 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:52.492134 6615 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:58:52.492165 6615 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:52.496431 6615 rpc_server.cc:307] RPC server started. Bound to: 127.6.117.195:41573
I20240502 01:58:52.496492 9154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.117.195:41573 every 8 connection(s)
I20240502 01:58:52.500613 9155 heartbeater.cc:347] Connected to a master server at 127.6.117.254:43557
I20240502 01:58:52.500715 9155 heartbeater.cc:464] Registering TS with master...
I20240502 01:58:52.500933 9155 heartbeater.cc:511] Master 127.6.117.254:43557 requested a full tablet report, sending...
I20240502 01:58:52.501538 8875 ts_manager.cc:194] Registered new tserver with Master: 5afe38d1059d4f6f80b1aef4810efaa8 (127.6.117.195:41573)
I20240502 01:58:52.502032 8875 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47104
I20240502 01:58:52.502439 6615 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.005612884s
I20240502 01:58:53.449328 9004 heartbeater.cc:503] Master 127.6.117.254:43557 was elected leader, sending a full tablet report...
I20240502 01:58:53.476784 9080 heartbeater.cc:503] Master 127.6.117.254:43557 was elected leader, sending a full tablet report...
I20240502 01:58:53.502912 9155 heartbeater.cc:503] Master 127.6.117.254:43557 was elected leader, sending a full tablet report...
I20240502 01:58:54.369971 8929 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:58:56.370352 8929 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:58:58.370697 8929 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:59:00.371019 8929 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:59:02.371359 8929 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:59:04.371704 8929 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:59:04.580142 6615 tablet_server.cc:178] TabletServer@127.6.117.193:0 shutting down...
I20240502 01:59:04.583755 6615 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:59:04.595728 6615 tablet_server.cc:194] TabletServer@127.6.117.193:0 shutdown complete.
I20240502 01:59:04.597642 6615 tablet_server.cc:178] TabletServer@127.6.117.194:0 shutting down...
I20240502 01:59:04.600881 6615 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:59:04.612654 6615 tablet_server.cc:194] TabletServer@127.6.117.194:0 shutdown complete.
I20240502 01:59:04.614334 6615 tablet_server.cc:178] TabletServer@127.6.117.195:0 shutting down...
I20240502 01:59:04.617648 6615 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:59:04.629320 6615 tablet_server.cc:194] TabletServer@127.6.117.195:0 shutdown complete.
I20240502 01:59:04.630983 6615 master.cc:537] Master@127.6.117.254:43557 shutting down...
I20240502 01:59:04.633810 6615 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 1433c29a836c48d8934cab39b6a12ce5 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:59:04.633972 6615 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 1433c29a836c48d8934cab39b6a12ce5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:04.634035 6615 tablet_replica.cc:330] T 00000000000000000000000000000000 P 1433c29a836c48d8934cab39b6a12ce5: stopping tablet replica
I20240502 01:59:04.646201 6615 master.cc:552] Master@127.6.117.254:43557 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfNoTablets (12316 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne
I20240502 01:59:04.651496 6615 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.6.117.254:42153
I20240502 01:59:04.651816 6615 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 01:59:04.654047 6615 server_base.cc:1034] running on GCE node
W20240502 01:59:04.654371 9159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:59:04.654685 9162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:59:04.655057 9160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:59:04.655241 6615 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:59:04.655285 6615 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:59:04.655304 6615 hybrid_clock.cc:648] HybridClock initialized: now 1714615144655303 us; error 0 us; skew 500 ppm
I20240502 01:59:04.661162 6615 webserver.cc:460] Webserver started at http://127.6.117.254:35633/ using document root <none> and password file <none>
I20240502 01:59:04.661307 6615 fs_manager.cc:359] Metadata directory not provided
I20240502 01:59:04.661357 6615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:59:04.661433 6615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:59:04.661816 6615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714615076189642-6615-0/minicluster-data/master-0-root/instance:
uuid: "366e960cf0f049b69bcc10b98f2aa5ff"
format_stamp: "Formatted at 2024-05-02 01:59:04 on dist-test-slave-89cm"
I20240502 01:59:04.663146 6615 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:04.663955 9167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:59:04.664217 6615 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:04.664327 6615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714615076189642-6615-0/minicluster-data/master-0-root
uuid: "366e960cf0f049b69bcc10b98f2aa5ff"
format_stamp: "Formatted at 2024-05-02 01:59:04 on dist-test-slave-89cm"
I20240502 01:59:04.664392 6615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714615076189642-6615-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714615076189642-6615-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714615076189642-6615-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:59:04.669134 6615 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:59:04.669394 6615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:59:04.673012 6615 rpc_server.cc:307] RPC server started. Bound to: 127.6.117.254:42153
I20240502 01:59:04.674168 9219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.117.254:42153 every 8 connection(s)
I20240502 01:59:04.674579 9220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:04.676223 9220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 366e960cf0f049b69bcc10b98f2aa5ff: Bootstrap starting.
I20240502 01:59:04.676946 9220 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 366e960cf0f049b69bcc10b98f2aa5ff: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:04.677806 9220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 366e960cf0f049b69bcc10b98f2aa5ff: No bootstrap required, opened a new log
I20240502 01:59:04.678123 9220 raft_consensus.cc:348] T 00000000000000000000000000000000 P 366e960cf0f049b69bcc10b98f2aa5ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "366e960cf0f049b69bcc10b98f2aa5ff" member_type: VOTER }
I20240502 01:59:04.678201 9220 raft_consensus.cc:374] T 00000000000000000000000000000000 P 366e960cf0f049b69bcc10b98f2aa5ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:04.678231 9220 raft_consensus.cc:729] T 00000000000000000000000000000000 P 366e960cf0f049b69bcc10b98f2aa5ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 366e960cf0f049b69bcc10b98f2aa5ff, State: Initialized, Role: FOLLOWER
I20240502 01:59:04.678323 9220 consensus_queue.cc:260] T 00000000000000000000000000000000 P 366e960cf0f049b69bcc10b98f2aa5ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "366e960cf0f049b69bcc10b98f2aa5ff" member_type: VOTER }
I20240502 01:59:04.678376 9220 raft_consensus.cc:388] T 00000000000000000000000000000000 P 366e960cf0f049b69bcc10b98f2aa5ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:59:04.678403 9220 raft_consensus.cc:482] T 00000000000000000000000000000000 P 366e960cf0f049b69bcc10b98f2aa5ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:59:04.678436 9220 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 366e960cf0f049b69bcc10b98f2aa5ff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:04.679008 9220 raft_consensus.cc:504] T 00000000000000000000000000000000 P 366e960cf0f049b69bcc10b98f2aa5ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "366e960cf0f049b69bcc10b98f2aa5ff" member_type: VOTER }
I20240502 01:59:04.679113 9220 leader_election.cc:302] T 00000000000000000000000000000000 P 366e960cf0f049b69bcc10b98f2aa5ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 366e960cf0f049b69bcc10b98f2aa5ff; no voters:
I20240502 01:59:04.679247 9220 leader_election.cc:288] T 00000000000000000000000000000000 P 366e960cf0f049b69bcc10b98f2aa5ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:59:04.679440 9223 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 366e960cf0f049b69bcc10b98f2aa5ff [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:04.679623 9223 raft_consensus.cc:686] T 00000000000000000000000000000000 P 366e960cf0f049b69bcc10b98f2aa5ff [term 1 LEADER]: Becoming Leader. State: Replica: 366e960cf0f049b69bcc10b98f2aa5ff, State: Running, Role: LEADER
I20240502 01:59:04.679649 9220 sys_catalog.cc:564] T 00000000000000000000000000000000 P 366e960cf0f049b69bcc10b98f2aa5ff [sys.catalog]: configured and running, proceeding with master startup.
I20240502 01:59:04.679747 9223 consensus_queue.cc:237] T 00000000000000000000000000000000 P 366e960cf0f049b69bcc10b98f2aa5ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "366e960cf0f049b69bcc10b98f2aa5ff" member_type: VOTER }
I20240502 01:59:04.680194 9225 sys_catalog.cc:455] T 00000000000000000000000000000000 P 366e960cf0f049b69bcc10b98f2aa5ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 366e960cf0f049b69bcc10b98f2aa5ff. Latest consensus state: current_term: 1 leader_uuid: "366e960cf0f049b69bcc10b98f2aa5ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "366e960cf0f049b69bcc10b98f2aa5ff" member_type: VOTER } }
I20240502 01:59:04.680171 9224 sys_catalog.cc:455] T 00000000000000000000000000000000 P 366e960cf0f049b69bcc10b98f2aa5ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "366e960cf0f049b69bcc10b98f2aa5ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "366e960cf0f049b69bcc10b98f2aa5ff" member_type: VOTER } }
I20240502 01:59:04.680280 9225 sys_catalog.cc:458] T 00000000000000000000000000000000 P 366e960cf0f049b69bcc10b98f2aa5ff [sys.catalog]: This master's current role is: LEADER
I20240502 01:59:04.680296 9224 sys_catalog.cc:458] T 00000000000000000000000000000000 P 366e960cf0f049b69bcc10b98f2aa5ff [sys.catalog]: This master's current role is: LEADER
I20240502 01:59:04.680784 9230 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 01:59:04.681602 9230 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 01:59:04.681782 6615 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 01:59:04.683220 9230 catalog_manager.cc:1347] Generated new cluster ID: b5965259610b47ba8a5317751197c453
I20240502 01:59:04.683269 9230 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 01:59:04.697669 9230 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 01:59:04.698174 9230 catalog_manager.cc:1504] Loading token signing keys...
I20240502 01:59:04.704146 9230 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 366e960cf0f049b69bcc10b98f2aa5ff: Generated new TSK 0
I20240502 01:59:04.704319 9230 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 01:59:04.713974 6615 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:59:04.715909 9241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:59:04.716290 9244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:59:04.716572 9242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:59:04.716982 6615 server_base.cc:1034] running on GCE node
I20240502 01:59:04.717105 6615 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:59:04.717139 6615 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:59:04.717154 6615 hybrid_clock.cc:648] HybridClock initialized: now 1714615144717153 us; error 0 us; skew 500 ppm
I20240502 01:59:04.717993 6615 webserver.cc:460] Webserver started at http://127.6.117.193:36349/ using document root <none> and password file <none>
I20240502 01:59:04.718124 6615 fs_manager.cc:359] Metadata directory not provided
I20240502 01:59:04.718166 6615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:59:04.718240 6615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:59:04.718662 6615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714615076189642-6615-0/minicluster-data/ts-0-root/instance:
uuid: "3d856bbd8a284de7b0e2df0d40a323ee"
format_stamp: "Formatted at 2024-05-02 01:59:04 on dist-test-slave-89cm"
I20240502 01:59:04.719944 6615 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240502 01:59:04.720782 9249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:59:04.720994 6615 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20240502 01:59:04.721053 6615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714615076189642-6615-0/minicluster-data/ts-0-root
uuid: "3d856bbd8a284de7b0e2df0d40a323ee"
format_stamp: "Formatted at 2024-05-02 01:59:04 on dist-test-slave-89cm"
I20240502 01:59:04.721118 6615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714615076189642-6615-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714615076189642-6615-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714615076189642-6615-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:59:04.740944 6615 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:59:04.741336 6615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:59:04.741634 6615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:59:04.742141 6615 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:59:04.742182 6615 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:59:04.742231 6615 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:59:04.742264 6615 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:59:04.746558 6615 rpc_server.cc:307] RPC server started. Bound to: 127.6.117.193:42299
I20240502 01:59:04.746600 9312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.117.193:42299 every 8 connection(s)
I20240502 01:59:04.747411 6615 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:59:04.749177 9317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:59:04.749473 9318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:59:04.749861 9320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:59:04.750295 6615 server_base.cc:1034] running on GCE node
I20240502 01:59:04.750494 6615 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:59:04.750537 6615 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:59:04.750560 6615 hybrid_clock.cc:648] HybridClock initialized: now 1714615144750559 us; error 0 us; skew 500 ppm
I20240502 01:59:04.751463 6615 webserver.cc:460] Webserver started at http://127.6.117.194:34559/ using document root <none> and password file <none>
I20240502 01:59:04.751622 6615 fs_manager.cc:359] Metadata directory not provided
I20240502 01:59:04.751685 6615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:59:04.751757 6615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:59:04.751839 9313 heartbeater.cc:347] Connected to a master server at 127.6.117.254:42153
I20240502 01:59:04.751935 9313 heartbeater.cc:464] Registering TS with master...
I20240502 01:59:04.752171 9313 heartbeater.cc:511] Master 127.6.117.254:42153 requested a full tablet report, sending...
I20240502 01:59:04.752219 6615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714615076189642-6615-0/minicluster-data/ts-1-root/instance:
uuid: "ee5c77a8fb434b3eab923a711a572596"
format_stamp: "Formatted at 2024-05-02 01:59:04 on dist-test-slave-89cm"
I20240502 01:59:04.752820 9184 ts_manager.cc:194] Registered new tserver with Master: 3d856bbd8a284de7b0e2df0d40a323ee (127.6.117.193:42299)
I20240502 01:59:04.753526 9184 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60798
I20240502 01:59:04.753767 6615 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:04.754637 9325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:59:04.754838 6615 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:04.754906 6615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714615076189642-6615-0/minicluster-data/ts-1-root
uuid: "ee5c77a8fb434b3eab923a711a572596"
format_stamp: "Formatted at 2024-05-02 01:59:04 on dist-test-slave-89cm"
I20240502 01:59:04.754974 6615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714615076189642-6615-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714615076189642-6615-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714615076189642-6615-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:59:04.768618 6615 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:59:04.768990 6615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:59:04.769258 6615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:59:04.769712 6615 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:59:04.769753 6615 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:59:04.769795 6615 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:59:04.769821 6615 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:59:04.773837 6615 rpc_server.cc:307] RPC server started. Bound to: 127.6.117.194:37953
I20240502 01:59:04.773875 9388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.117.194:37953 every 8 connection(s)
I20240502 01:59:04.774560 6615 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:59:04.776423 9392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:59:04.776742 9393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:59:04.777065 9395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:59:04.777845 6615 server_base.cc:1034] running on GCE node
I20240502 01:59:04.778026 6615 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:59:04.778074 6615 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:59:04.778088 6615 hybrid_clock.cc:648] HybridClock initialized: now 1714615144778088 us; error 0 us; skew 500 ppm
I20240502 01:59:04.778869 9389 heartbeater.cc:347] Connected to a master server at 127.6.117.254:42153
I20240502 01:59:04.778957 9389 heartbeater.cc:464] Registering TS with master...
I20240502 01:59:04.778997 6615 webserver.cc:460] Webserver started at http://127.6.117.195:42919/ using document root <none> and password file <none>
I20240502 01:59:04.779145 6615 fs_manager.cc:359] Metadata directory not provided
I20240502 01:59:04.779165 9389 heartbeater.cc:511] Master 127.6.117.254:42153 requested a full tablet report, sending...
I20240502 01:59:04.779191 6615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:59:04.779270 6615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:59:04.779695 6615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714615076189642-6615-0/minicluster-data/ts-2-root/instance:
uuid: "926089692ec840a5b0ad2932addccaae"
format_stamp: "Formatted at 2024-05-02 01:59:04 on dist-test-slave-89cm"
I20240502 01:59:04.779722 9184 ts_manager.cc:194] Registered new tserver with Master: ee5c77a8fb434b3eab923a711a572596 (127.6.117.194:37953)
I20240502 01:59:04.780318 9184 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60806
I20240502 01:59:04.781167 6615 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240502 01:59:04.781975 9400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:59:04.782176 6615 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:04.782243 6615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714615076189642-6615-0/minicluster-data/ts-2-root
uuid: "926089692ec840a5b0ad2932addccaae"
format_stamp: "Formatted at 2024-05-02 01:59:04 on dist-test-slave-89cm"
I20240502 01:59:04.782305 6615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714615076189642-6615-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714615076189642-6615-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714615076189642-6615-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:59:04.798622 6615 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:59:04.799018 6615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:59:04.799305 6615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:59:04.799726 6615 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:59:04.799765 6615 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:59:04.799805 6615 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:59:04.799831 6615 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:59:04.804073 6615 rpc_server.cc:307] RPC server started. Bound to: 127.6.117.195:46177
I20240502 01:59:04.804123 9463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.117.195:46177 every 8 connection(s)
I20240502 01:59:04.804821 6615 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:59:04.806739 9467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:59:04.807063 9468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:59:04.807385 9470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:59:04.807818 6615 server_base.cc:1034] running on GCE node
I20240502 01:59:04.807993 6615 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:59:04.808033 6615 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:59:04.808053 6615 hybrid_clock.cc:648] HybridClock initialized: now 1714615144808052 us; error 0 us; skew 500 ppm
I20240502 01:59:04.808773 9464 heartbeater.cc:347] Connected to a master server at 127.6.117.254:42153
I20240502 01:59:04.808867 9464 heartbeater.cc:464] Registering TS with master...
I20240502 01:59:04.809024 6615 webserver.cc:460] Webserver started at http://127.6.117.196:37499/ using document root <none> and password file <none>
I20240502 01:59:04.809078 9464 heartbeater.cc:511] Master 127.6.117.254:42153 requested a full tablet report, sending...
I20240502 01:59:04.809177 6615 fs_manager.cc:359] Metadata directory not provided
I20240502 01:59:04.809226 6615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:59:04.809298 6615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:59:04.809643 9184 ts_manager.cc:194] Registered new tserver with Master: 926089692ec840a5b0ad2932addccaae (127.6.117.195:46177)
I20240502 01:59:04.809701 6615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714615076189642-6615-0/minicluster-data/ts-3-root/instance:
uuid: "947290d4724548afb1dcd97e97dd078f"
format_stamp: "Formatted at 2024-05-02 01:59:04 on dist-test-slave-89cm"
I20240502 01:59:04.810111 9184 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60820
I20240502 01:59:04.811102 6615 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240502 01:59:04.811929 9475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:59:04.812155 6615 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240502 01:59:04.812216 6615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714615076189642-6615-0/minicluster-data/ts-3-root
uuid: "947290d4724548afb1dcd97e97dd078f"
format_stamp: "Formatted at 2024-05-02 01:59:04 on dist-test-slave-89cm"
I20240502 01:59:04.812302 6615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714615076189642-6615-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714615076189642-6615-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714615076189642-6615-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:59:04.821902 6615 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:59:04.822261 6615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:59:04.822579 6615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:59:04.823010 6615 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:59:04.823048 6615 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:59:04.823087 6615 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:59:04.823114 6615 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:59:04.827265 6615 rpc_server.cc:307] RPC server started. Bound to: 127.6.117.196:41255
I20240502 01:59:04.827322 9538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.117.196:41255 every 8 connection(s)
I20240502 01:59:04.831096 9539 heartbeater.cc:347] Connected to a master server at 127.6.117.254:42153
I20240502 01:59:04.831177 9539 heartbeater.cc:464] Registering TS with master...
I20240502 01:59:04.831353 9539 heartbeater.cc:511] Master 127.6.117.254:42153 requested a full tablet report, sending...
I20240502 01:59:04.831871 9184 ts_manager.cc:194] Registered new tserver with Master: 947290d4724548afb1dcd97e97dd078f (127.6.117.196:41255)
I20240502 01:59:04.832104 6615 internal_mini_cluster.cc:371] 4 TS(s) registered with all masters after 0.00447109s
I20240502 01:59:04.832397 9184 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60836
I20240502 01:59:05.754386 9313 heartbeater.cc:503] Master 127.6.117.254:42153 was elected leader, sending a full tablet report...
I20240502 01:59:05.781167 9389 heartbeater.cc:503] Master 127.6.117.254:42153 was elected leader, sending a full tablet report...
I20240502 01:59:05.810905 9464 heartbeater.cc:503] Master 127.6.117.254:42153 was elected leader, sending a full tablet report...
I20240502 01:59:05.833179 9539 heartbeater.cc:503] Master 127.6.117.254:42153 was elected leader, sending a full tablet report...
I20240502 01:59:05.862149 6615 test_util.cc:268] Using random seed: -223210494
I20240502 01:59:05.869160 9184 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60856:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 01:59:05.878688 9428 tablet_service.cc:1467] Processing CreateTablet for tablet da308e670ce64e32913ed271591daebc (DEFAULT_TABLE table=test-workload [id=eedb1d45ad194bdcb10fae7243f3c6f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:05.878754 9277 tablet_service.cc:1467] Processing CreateTablet for tablet da308e670ce64e32913ed271591daebc (DEFAULT_TABLE table=test-workload [id=eedb1d45ad194bdcb10fae7243f3c6f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:05.878688 9353 tablet_service.cc:1467] Processing CreateTablet for tablet da308e670ce64e32913ed271591daebc (DEFAULT_TABLE table=test-workload [id=eedb1d45ad194bdcb10fae7243f3c6f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:05.879035 9277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da308e670ce64e32913ed271591daebc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:05.879029 9353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da308e670ce64e32913ed271591daebc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:05.879029 9428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da308e670ce64e32913ed271591daebc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:05.881294 9560 tablet_bootstrap.cc:492] T da308e670ce64e32913ed271591daebc P 3d856bbd8a284de7b0e2df0d40a323ee: Bootstrap starting.
I20240502 01:59:05.881608 9561 tablet_bootstrap.cc:492] T da308e670ce64e32913ed271591daebc P 926089692ec840a5b0ad2932addccaae: Bootstrap starting.
I20240502 01:59:05.882164 9560 tablet_bootstrap.cc:654] T da308e670ce64e32913ed271591daebc P 3d856bbd8a284de7b0e2df0d40a323ee: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:05.882375 9561 tablet_bootstrap.cc:654] T da308e670ce64e32913ed271591daebc P 926089692ec840a5b0ad2932addccaae: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:05.882964 9562 tablet_bootstrap.cc:492] T da308e670ce64e32913ed271591daebc P ee5c77a8fb434b3eab923a711a572596: Bootstrap starting.
I20240502 01:59:05.883169 9560 tablet_bootstrap.cc:492] T da308e670ce64e32913ed271591daebc P 3d856bbd8a284de7b0e2df0d40a323ee: No bootstrap required, opened a new log
I20240502 01:59:05.883253 9560 ts_tablet_manager.cc:1367] T da308e670ce64e32913ed271591daebc P 3d856bbd8a284de7b0e2df0d40a323ee: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:59:05.883531 9561 tablet_bootstrap.cc:492] T da308e670ce64e32913ed271591daebc P 926089692ec840a5b0ad2932addccaae: No bootstrap required, opened a new log
I20240502 01:59:05.883613 9561 ts_tablet_manager.cc:1367] T da308e670ce64e32913ed271591daebc P 926089692ec840a5b0ad2932addccaae: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:59:05.883648 9560 raft_consensus.cc:348] T da308e670ce64e32913ed271591daebc P 3d856bbd8a284de7b0e2df0d40a323ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d856bbd8a284de7b0e2df0d40a323ee" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 42299 } } peers { permanent_uuid: "926089692ec840a5b0ad2932addccaae" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 46177 } } peers { permanent_uuid: "ee5c77a8fb434b3eab923a711a572596" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 37953 } }
I20240502 01:59:05.883756 9560 raft_consensus.cc:374] T da308e670ce64e32913ed271591daebc P 3d856bbd8a284de7b0e2df0d40a323ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:05.883809 9560 raft_consensus.cc:729] T da308e670ce64e32913ed271591daebc P 3d856bbd8a284de7b0e2df0d40a323ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d856bbd8a284de7b0e2df0d40a323ee, State: Initialized, Role: FOLLOWER
I20240502 01:59:05.883852 9562 tablet_bootstrap.cc:654] T da308e670ce64e32913ed271591daebc P ee5c77a8fb434b3eab923a711a572596: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:05.883960 9560 consensus_queue.cc:260] T da308e670ce64e32913ed271591daebc P 3d856bbd8a284de7b0e2df0d40a323ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d856bbd8a284de7b0e2df0d40a323ee" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 42299 } } peers { permanent_uuid: "926089692ec840a5b0ad2932addccaae" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 46177 } } peers { permanent_uuid: "ee5c77a8fb434b3eab923a711a572596" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 37953 } }
I20240502 01:59:05.884002 9561 raft_consensus.cc:348] T da308e670ce64e32913ed271591daebc P 926089692ec840a5b0ad2932addccaae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d856bbd8a284de7b0e2df0d40a323ee" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 42299 } } peers { permanent_uuid: "926089692ec840a5b0ad2932addccaae" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 46177 } } peers { permanent_uuid: "ee5c77a8fb434b3eab923a711a572596" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 37953 } }
I20240502 01:59:05.884243 9561 raft_consensus.cc:374] T da308e670ce64e32913ed271591daebc P 926089692ec840a5b0ad2932addccaae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:05.884356 9561 raft_consensus.cc:729] T da308e670ce64e32913ed271591daebc P 926089692ec840a5b0ad2932addccaae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 926089692ec840a5b0ad2932addccaae, State: Initialized, Role: FOLLOWER
I20240502 01:59:05.884435 9560 ts_tablet_manager.cc:1398] T da308e670ce64e32913ed271591daebc P 3d856bbd8a284de7b0e2df0d40a323ee: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:05.884549 9561 consensus_queue.cc:260] T da308e670ce64e32913ed271591daebc P 926089692ec840a5b0ad2932addccaae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d856bbd8a284de7b0e2df0d40a323ee" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 42299 } } peers { permanent_uuid: "926089692ec840a5b0ad2932addccaae" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 46177 } } peers { permanent_uuid: "ee5c77a8fb434b3eab923a711a572596" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 37953 } }
I20240502 01:59:05.884877 9562 tablet_bootstrap.cc:492] T da308e670ce64e32913ed271591daebc P ee5c77a8fb434b3eab923a711a572596: No bootstrap required, opened a new log
I20240502 01:59:05.884953 9562 ts_tablet_manager.cc:1367] T da308e670ce64e32913ed271591daebc P ee5c77a8fb434b3eab923a711a572596: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:59:05.884974 9561 ts_tablet_manager.cc:1398] T da308e670ce64e32913ed271591daebc P 926089692ec840a5b0ad2932addccaae: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:59:05.885289 9562 raft_consensus.cc:348] T da308e670ce64e32913ed271591daebc P ee5c77a8fb434b3eab923a711a572596 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d856bbd8a284de7b0e2df0d40a323ee" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 42299 } } peers { permanent_uuid: "926089692ec840a5b0ad2932addccaae" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 46177 } } peers { permanent_uuid: "ee5c77a8fb434b3eab923a711a572596" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 37953 } }
I20240502 01:59:05.885370 9562 raft_consensus.cc:374] T da308e670ce64e32913ed271591daebc P ee5c77a8fb434b3eab923a711a572596 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:05.885398 9562 raft_consensus.cc:729] T da308e670ce64e32913ed271591daebc P ee5c77a8fb434b3eab923a711a572596 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee5c77a8fb434b3eab923a711a572596, State: Initialized, Role: FOLLOWER
I20240502 01:59:05.885507 9562 consensus_queue.cc:260] T da308e670ce64e32913ed271591daebc P ee5c77a8fb434b3eab923a711a572596 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d856bbd8a284de7b0e2df0d40a323ee" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 42299 } } peers { permanent_uuid: "926089692ec840a5b0ad2932addccaae" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 46177 } } peers { permanent_uuid: "ee5c77a8fb434b3eab923a711a572596" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 37953 } }
I20240502 01:59:05.885824 9562 ts_tablet_manager.cc:1398] T da308e670ce64e32913ed271591daebc P ee5c77a8fb434b3eab923a711a572596: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:05.948881 9568 raft_consensus.cc:482] T da308e670ce64e32913ed271591daebc P ee5c77a8fb434b3eab923a711a572596 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:05.949031 9568 raft_consensus.cc:504] T da308e670ce64e32913ed271591daebc P ee5c77a8fb434b3eab923a711a572596 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d856bbd8a284de7b0e2df0d40a323ee" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 42299 } } peers { permanent_uuid: "926089692ec840a5b0ad2932addccaae" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 46177 } } peers { permanent_uuid: "ee5c77a8fb434b3eab923a711a572596" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 37953 } }
I20240502 01:59:05.949539 9568 leader_election.cc:288] T da308e670ce64e32913ed271591daebc P ee5c77a8fb434b3eab923a711a572596 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d856bbd8a284de7b0e2df0d40a323ee (127.6.117.193:42299), 926089692ec840a5b0ad2932addccaae (127.6.117.195:46177)
I20240502 01:59:05.952924 9287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da308e670ce64e32913ed271591daebc" candidate_uuid: "ee5c77a8fb434b3eab923a711a572596" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d856bbd8a284de7b0e2df0d40a323ee" is_pre_election: true
I20240502 01:59:05.952988 9438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da308e670ce64e32913ed271591daebc" candidate_uuid: "ee5c77a8fb434b3eab923a711a572596" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "926089692ec840a5b0ad2932addccaae" is_pre_election: true
I20240502 01:59:05.953063 9287 raft_consensus.cc:2454] T da308e670ce64e32913ed271591daebc P 3d856bbd8a284de7b0e2df0d40a323ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee5c77a8fb434b3eab923a711a572596 in term 0.
I20240502 01:59:05.953122 9438 raft_consensus.cc:2454] T da308e670ce64e32913ed271591daebc P 926089692ec840a5b0ad2932addccaae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee5c77a8fb434b3eab923a711a572596 in term 0.
I20240502 01:59:05.953362 9329 leader_election.cc:302] T da308e670ce64e32913ed271591daebc P ee5c77a8fb434b3eab923a711a572596 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d856bbd8a284de7b0e2df0d40a323ee, ee5c77a8fb434b3eab923a711a572596; no voters:
I20240502 01:59:05.953536 9568 raft_consensus.cc:2781] T da308e670ce64e32913ed271591daebc P ee5c77a8fb434b3eab923a711a572596 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:05.953595 9568 raft_consensus.cc:482] T da308e670ce64e32913ed271591daebc P ee5c77a8fb434b3eab923a711a572596 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:05.953641 9568 raft_consensus.cc:3037] T da308e670ce64e32913ed271591daebc P ee5c77a8fb434b3eab923a711a572596 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:05.954432 9568 raft_consensus.cc:504] T da308e670ce64e32913ed271591daebc P ee5c77a8fb434b3eab923a711a572596 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d856bbd8a284de7b0e2df0d40a323ee" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 42299 } } peers { permanent_uuid: "926089692ec840a5b0ad2932addccaae" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 46177 } } peers { permanent_uuid: "ee5c77a8fb434b3eab923a711a572596" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 37953 } }
I20240502 01:59:05.954783 9568 leader_election.cc:288] T da308e670ce64e32913ed271591daebc P ee5c77a8fb434b3eab923a711a572596 [CANDIDATE]: Term 1 election: Requested vote from peers 3d856bbd8a284de7b0e2df0d40a323ee (127.6.117.193:42299), 926089692ec840a5b0ad2932addccaae (127.6.117.195:46177)
I20240502 01:59:05.955031 9287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da308e670ce64e32913ed271591daebc" candidate_uuid: "ee5c77a8fb434b3eab923a711a572596" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d856bbd8a284de7b0e2df0d40a323ee"
I20240502 01:59:05.955031 9438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da308e670ce64e32913ed271591daebc" candidate_uuid: "ee5c77a8fb434b3eab923a711a572596" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "926089692ec840a5b0ad2932addccaae"
I20240502 01:59:05.955159 9438 raft_consensus.cc:3037] T da308e670ce64e32913ed271591daebc P 926089692ec840a5b0ad2932addccaae [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:05.955159 9287 raft_consensus.cc:3037] T da308e670ce64e32913ed271591daebc P 3d856bbd8a284de7b0e2df0d40a323ee [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:05.956100 9287 raft_consensus.cc:2454] T da308e670ce64e32913ed271591daebc P 3d856bbd8a284de7b0e2df0d40a323ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee5c77a8fb434b3eab923a711a572596 in term 1.
I20240502 01:59:05.956100 9438 raft_consensus.cc:2454] T da308e670ce64e32913ed271591daebc P 926089692ec840a5b0ad2932addccaae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee5c77a8fb434b3eab923a711a572596 in term 1.
I20240502 01:59:05.956378 9326 leader_election.cc:302] T da308e670ce64e32913ed271591daebc P ee5c77a8fb434b3eab923a711a572596 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 926089692ec840a5b0ad2932addccaae, ee5c77a8fb434b3eab923a711a572596; no voters:
I20240502 01:59:05.956569 9568 raft_consensus.cc:2781] T da308e670ce64e32913ed271591daebc P ee5c77a8fb434b3eab923a711a572596 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:05.956749 9568 raft_consensus.cc:686] T da308e670ce64e32913ed271591daebc P ee5c77a8fb434b3eab923a711a572596 [term 1 LEADER]: Becoming Leader. State: Replica: ee5c77a8fb434b3eab923a711a572596, State: Running, Role: LEADER
I20240502 01:59:05.956880 9568 consensus_queue.cc:237] T da308e670ce64e32913ed271591daebc P ee5c77a8fb434b3eab923a711a572596 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d856bbd8a284de7b0e2df0d40a323ee" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 42299 } } peers { permanent_uuid: "926089692ec840a5b0ad2932addccaae" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 46177 } } peers { permanent_uuid: "ee5c77a8fb434b3eab923a711a572596" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 37953 } }
I20240502 01:59:05.958187 9184 catalog_manager.cc:5484] T da308e670ce64e32913ed271591daebc P ee5c77a8fb434b3eab923a711a572596 reported cstate change: term changed from 0 to 1, leader changed from <none> to ee5c77a8fb434b3eab923a711a572596 (127.6.117.194). New cstate: current_term: 1 leader_uuid: "ee5c77a8fb434b3eab923a711a572596" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d856bbd8a284de7b0e2df0d40a323ee" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 42299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "926089692ec840a5b0ad2932addccaae" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 46177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee5c77a8fb434b3eab923a711a572596" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 37953 } health_report { overall_health: HEALTHY } } }
W20240502 01:59:05.998277 9314 tablet.cc:2369] T da308e670ce64e32913ed271591daebc P 3d856bbd8a284de7b0e2df0d40a323ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:59:06.366544 9568 consensus_queue.cc:1035] T da308e670ce64e32913ed271591daebc P ee5c77a8fb434b3eab923a711a572596 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d856bbd8a284de7b0e2df0d40a323ee" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 42299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:06.369367 9568 consensus_queue.cc:1035] T da308e670ce64e32913ed271591daebc P ee5c77a8fb434b3eab923a711a572596 [LEADER]: Connected to new peer: Peer: permanent_uuid: "926089692ec840a5b0ad2932addccaae" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 46177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:06.681547 9238 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:59:06.681854 9238 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:59:06.681922 9238 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:59:08.682219 9238 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:59:08.682525 9238 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:59:08.682590 9238 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:59:10.682870 9238 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:59:10.683178 9238 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:59:10.683245 9238 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:59:12.683504 9238 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:59:12.683802 9238 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:59:12.683869 9238 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:59:14.684157 9238 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:59:14.684474 9238 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:59:14.684540 9238 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:59:16.684864 9238 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:59:16.685163 9238 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:59:16.685218 9238 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:59:17.021260 6615 tablet_server.cc:178] TabletServer@127.6.117.193:0 shutting down...
I20240502 01:59:17.024538 6615 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:59:17.024729 6615 tablet_replica.cc:330] T da308e670ce64e32913ed271591daebc P 3d856bbd8a284de7b0e2df0d40a323ee: stopping tablet replica
I20240502 01:59:17.024853 6615 raft_consensus.cc:2229] T da308e670ce64e32913ed271591daebc P 3d856bbd8a284de7b0e2df0d40a323ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:17.024971 6615 raft_consensus.cc:2258] T da308e670ce64e32913ed271591daebc P 3d856bbd8a284de7b0e2df0d40a323ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:17.037529 6615 tablet_server.cc:194] TabletServer@127.6.117.193:0 shutdown complete.
I20240502 01:59:17.039444 6615 tablet_server.cc:178] TabletServer@127.6.117.194:0 shutting down...
I20240502 01:59:17.042639 6615 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:59:17.042821 6615 tablet_replica.cc:330] T da308e670ce64e32913ed271591daebc P ee5c77a8fb434b3eab923a711a572596: stopping tablet replica
I20240502 01:59:17.042929 6615 raft_consensus.cc:2229] T da308e670ce64e32913ed271591daebc P ee5c77a8fb434b3eab923a711a572596 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:59:17.043130 6615 raft_consensus.cc:2258] T da308e670ce64e32913ed271591daebc P ee5c77a8fb434b3eab923a711a572596 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:17.055662 6615 tablet_server.cc:194] TabletServer@127.6.117.194:0 shutdown complete.
I20240502 01:59:17.057686 6615 tablet_server.cc:178] TabletServer@127.6.117.195:0 shutting down...
I20240502 01:59:17.060709 6615 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:59:17.060897 6615 tablet_replica.cc:330] T da308e670ce64e32913ed271591daebc P 926089692ec840a5b0ad2932addccaae: stopping tablet replica
I20240502 01:59:17.061007 6615 raft_consensus.cc:2229] T da308e670ce64e32913ed271591daebc P 926089692ec840a5b0ad2932addccaae [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:17.061111 6615 raft_consensus.cc:2258] T da308e670ce64e32913ed271591daebc P 926089692ec840a5b0ad2932addccaae [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:17.073604 6615 tablet_server.cc:194] TabletServer@127.6.117.195:0 shutdown complete.
I20240502 01:59:17.075544 6615 tablet_server.cc:178] TabletServer@127.6.117.196:0 shutting down...
I20240502 01:59:17.078784 6615 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:59:17.090556 6615 tablet_server.cc:194] TabletServer@127.6.117.196:0 shutdown complete.
I20240502 01:59:17.092379 6615 master.cc:537] Master@127.6.117.254:42153 shutting down...
I20240502 01:59:17.095487 6615 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 366e960cf0f049b69bcc10b98f2aa5ff [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:59:17.095649 6615 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 366e960cf0f049b69bcc10b98f2aa5ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:17.095710 6615 tablet_replica.cc:330] T 00000000000000000000000000000000 P 366e960cf0f049b69bcc10b98f2aa5ff: stopping tablet replica
I20240502 01:59:17.108147 6615 master.cc:552] Master@127.6.117.254:42153 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne (12462 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy
I20240502 01:59:17.114382 6615 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.6.117.254:33303
I20240502 01:59:17.114710 6615 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:59:17.116911 9580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:59:17.117233 9582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:59:17.117591 9579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:59:17.117992 6615 server_base.cc:1034] running on GCE node
I20240502 01:59:17.118122 6615 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:59:17.118158 6615 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:59:17.118172 6615 hybrid_clock.cc:648] HybridClock initialized: now 1714615157118172 us; error 0 us; skew 500 ppm
I20240502 01:59:17.119001 6615 webserver.cc:460] Webserver started at http://127.6.117.254:37641/ using document root <none> and password file <none>
I20240502 01:59:17.119122 6615 fs_manager.cc:359] Metadata directory not provided
I20240502 01:59:17.119168 6615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:59:17.119226 6615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:59:17.119580 6615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714615076189642-6615-0/minicluster-data/master-0-root/instance:
uuid: "78a29320f19d4bea835089f246fbeffa"
format_stamp: "Formatted at 2024-05-02 01:59:17 on dist-test-slave-89cm"
I20240502 01:59:17.120977 6615 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240502 01:59:17.121901 9587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:59:17.122200 6615 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:17.122263 6615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714615076189642-6615-0/minicluster-data/master-0-root
uuid: "78a29320f19d4bea835089f246fbeffa"
format_stamp: "Formatted at 2024-05-02 01:59:17 on dist-test-slave-89cm"
I20240502 01:59:17.122323 6615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714615076189642-6615-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714615076189642-6615-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714615076189642-6615-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:59:17.131006 6615 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:59:17.131321 6615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:59:17.135304 6615 rpc_server.cc:307] RPC server started. Bound to: 127.6.117.254:33303
I20240502 01:59:17.137173 9639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.117.254:33303 every 8 connection(s)
I20240502 01:59:17.137619 9640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:17.139392 9640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78a29320f19d4bea835089f246fbeffa: Bootstrap starting.
I20240502 01:59:17.140128 9640 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 78a29320f19d4bea835089f246fbeffa: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:17.141099 9640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78a29320f19d4bea835089f246fbeffa: No bootstrap required, opened a new log
I20240502 01:59:17.141469 9640 raft_consensus.cc:348] T 00000000000000000000000000000000 P 78a29320f19d4bea835089f246fbeffa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78a29320f19d4bea835089f246fbeffa" member_type: VOTER }
I20240502 01:59:17.141557 9640 raft_consensus.cc:374] T 00000000000000000000000000000000 P 78a29320f19d4bea835089f246fbeffa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:17.141592 9640 raft_consensus.cc:729] T 00000000000000000000000000000000 P 78a29320f19d4bea835089f246fbeffa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78a29320f19d4bea835089f246fbeffa, State: Initialized, Role: FOLLOWER
I20240502 01:59:17.141731 9640 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78a29320f19d4bea835089f246fbeffa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78a29320f19d4bea835089f246fbeffa" member_type: VOTER }
I20240502 01:59:17.141796 9640 raft_consensus.cc:388] T 00000000000000000000000000000000 P 78a29320f19d4bea835089f246fbeffa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:59:17.141829 9640 raft_consensus.cc:482] T 00000000000000000000000000000000 P 78a29320f19d4bea835089f246fbeffa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:59:17.141871 9640 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 78a29320f19d4bea835089f246fbeffa [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:17.142508 9640 raft_consensus.cc:504] T 00000000000000000000000000000000 P 78a29320f19d4bea835089f246fbeffa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78a29320f19d4bea835089f246fbeffa" member_type: VOTER }
I20240502 01:59:17.142616 9640 leader_election.cc:302] T 00000000000000000000000000000000 P 78a29320f19d4bea835089f246fbeffa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 78a29320f19d4bea835089f246fbeffa; no voters:
I20240502 01:59:17.142779 9640 leader_election.cc:288] T 00000000000000000000000000000000 P 78a29320f19d4bea835089f246fbeffa [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:59:17.142881 9643 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 78a29320f19d4bea835089f246fbeffa [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:17.143060 9643 raft_consensus.cc:686] T 00000000000000000000000000000000 P 78a29320f19d4bea835089f246fbeffa [term 1 LEADER]: Becoming Leader. State: Replica: 78a29320f19d4bea835089f246fbeffa, State: Running, Role: LEADER
I20240502 01:59:17.143172 9640 sys_catalog.cc:564] T 00000000000000000000000000000000 P 78a29320f19d4bea835089f246fbeffa [sys.catalog]: configured and running, proceeding with master startup.
I20240502 01:59:17.143184 9643 consensus_queue.cc:237] T 00000000000000000000000000000000 P 78a29320f19d4bea835089f246fbeffa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78a29320f19d4bea835089f246fbeffa" member_type: VOTER }
I20240502 01:59:17.143613 9644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78a29320f19d4bea835089f246fbeffa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "78a29320f19d4bea835089f246fbeffa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78a29320f19d4bea835089f246fbeffa" member_type: VOTER } }
I20240502 01:59:17.143637 9645 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78a29320f19d4bea835089f246fbeffa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78a29320f19d4bea835089f246fbeffa. Latest consensus state: current_term: 1 leader_uuid: "78a29320f19d4bea835089f246fbeffa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78a29320f19d4bea835089f246fbeffa" member_type: VOTER } }
I20240502 01:59:17.143712 9644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78a29320f19d4bea835089f246fbeffa [sys.catalog]: This master's current role is: LEADER
I20240502 01:59:17.143719 9645 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78a29320f19d4bea835089f246fbeffa [sys.catalog]: This master's current role is: LEADER
I20240502 01:59:17.143975 9648 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 01:59:17.144806 9648 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 01:59:17.145095 6615 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 01:59:17.146436 9648 catalog_manager.cc:1347] Generated new cluster ID: 99c862099a834a898d9ec1ddc2e71d99
I20240502 01:59:17.146488 9648 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 01:59:17.170342 9648 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 01:59:17.170910 9648 catalog_manager.cc:1504] Loading token signing keys...
I20240502 01:59:17.180444 9648 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 78a29320f19d4bea835089f246fbeffa: Generated new TSK 0
I20240502 01:59:17.180622 9648 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 01:59:17.209586 6615 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:59:17.211709 9661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:59:17.212069 9664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:59:17.212443 9662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:59:17.212884 6615 server_base.cc:1034] running on GCE node
I20240502 01:59:17.213016 6615 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:59:17.213047 6615 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:59:17.213063 6615 hybrid_clock.cc:648] HybridClock initialized: now 1714615157213062 us; error 0 us; skew 500 ppm
I20240502 01:59:17.213959 6615 webserver.cc:460] Webserver started at http://127.6.117.193:34877/ using document root <none> and password file <none>
I20240502 01:59:17.214092 6615 fs_manager.cc:359] Metadata directory not provided
I20240502 01:59:17.214133 6615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:59:17.214187 6615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:59:17.214521 6615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714615076189642-6615-0/minicluster-data/ts-0-root/instance:
uuid: "86b8193cab1b4138a57d00fc6f8927b1"
format_stamp: "Formatted at 2024-05-02 01:59:17 on dist-test-slave-89cm"
I20240502 01:59:17.215809 6615 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:17.216655 9669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:59:17.216871 6615 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20240502 01:59:17.216933 6615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714615076189642-6615-0/minicluster-data/ts-0-root
uuid: "86b8193cab1b4138a57d00fc6f8927b1"
format_stamp: "Formatted at 2024-05-02 01:59:17 on dist-test-slave-89cm"
I20240502 01:59:17.216997 6615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714615076189642-6615-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714615076189642-6615-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714615076189642-6615-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:59:17.235065 6615 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:59:17.235433 6615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:59:17.235729 6615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:59:17.236188 6615 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:59:17.236229 6615 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:59:17.236292 6615 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:59:17.236321 6615 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:59:17.240211 6615 rpc_server.cc:307] RPC server started. Bound to: 127.6.117.193:40583
I20240502 01:59:17.240237 9732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.117.193:40583 every 8 connection(s)
I20240502 01:59:17.240988 6615 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:59:17.242813 9738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:59:17.243193 9740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:59:17.243492 9737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:59:17.243898 6615 server_base.cc:1034] running on GCE node
I20240502 01:59:17.244046 6615 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:59:17.244086 6615 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:59:17.244099 6615 hybrid_clock.cc:648] HybridClock initialized: now 1714615157244098 us; error 0 us; skew 500 ppm
I20240502 01:59:17.245196 6615 webserver.cc:460] Webserver started at http://127.6.117.194:43619/ using document root <none> and password file <none>
I20240502 01:59:17.245347 6615 fs_manager.cc:359] Metadata directory not provided
I20240502 01:59:17.245406 6615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:59:17.245483 6615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:59:17.245903 6615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714615076189642-6615-0/minicluster-data/ts-1-root/instance:
uuid: "c29ceaea1f7c4cc89ade5b8c12952856"
format_stamp: "Formatted at 2024-05-02 01:59:17 on dist-test-slave-89cm"
I20240502 01:59:17.246311 9733 heartbeater.cc:347] Connected to a master server at 127.6.117.254:33303
I20240502 01:59:17.246410 9733 heartbeater.cc:464] Registering TS with master...
I20240502 01:59:17.246613 9733 heartbeater.cc:511] Master 127.6.117.254:33303 requested a full tablet report, sending...
I20240502 01:59:17.247268 9604 ts_manager.cc:194] Registered new tserver with Master: 86b8193cab1b4138a57d00fc6f8927b1 (127.6.117.193:40583)
I20240502 01:59:17.247507 6615 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240502 01:59:17.248276 9604 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43328
I20240502 01:59:17.248558 9745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:59:17.248847 6615 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:17.248934 6615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714615076189642-6615-0/minicluster-data/ts-1-root
uuid: "c29ceaea1f7c4cc89ade5b8c12952856"
format_stamp: "Formatted at 2024-05-02 01:59:17 on dist-test-slave-89cm"
I20240502 01:59:17.249012 6615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714615076189642-6615-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714615076189642-6615-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714615076189642-6615-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:59:17.262131 6615 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:59:17.262461 6615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:59:17.262722 6615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:59:17.263192 6615 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:59:17.263245 6615 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:59:17.263289 6615 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:59:17.263353 6615 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:59:17.267987 6615 rpc_server.cc:307] RPC server started. Bound to: 127.6.117.194:34609
I20240502 01:59:17.268038 9808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.117.194:34609 every 8 connection(s)
I20240502 01:59:17.268739 6615 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:59:17.270597 9813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:59:17.270859 9812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:59:17.271255 9815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:59:17.271700 6615 server_base.cc:1034] running on GCE node
I20240502 01:59:17.271836 6615 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:59:17.271871 6615 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:59:17.271890 6615 hybrid_clock.cc:648] HybridClock initialized: now 1714615157271889 us; error 0 us; skew 500 ppm
I20240502 01:59:17.272817 6615 webserver.cc:460] Webserver started at http://127.6.117.195:43367/ using document root <none> and password file <none>
I20240502 01:59:17.272945 9809 heartbeater.cc:347] Connected to a master server at 127.6.117.254:33303
I20240502 01:59:17.272961 6615 fs_manager.cc:359] Metadata directory not provided
I20240502 01:59:17.273025 6615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:59:17.273044 9809 heartbeater.cc:464] Registering TS with master...
I20240502 01:59:17.273097 6615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:59:17.273274 9809 heartbeater.cc:511] Master 127.6.117.254:33303 requested a full tablet report, sending...
I20240502 01:59:17.273562 6615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714615076189642-6615-0/minicluster-data/ts-2-root/instance:
uuid: "6b35082b827b4212a4f4098dac93a5a6"
format_stamp: "Formatted at 2024-05-02 01:59:17 on dist-test-slave-89cm"
I20240502 01:59:17.273854 9604 ts_manager.cc:194] Registered new tserver with Master: c29ceaea1f7c4cc89ade5b8c12952856 (127.6.117.194:34609)
I20240502 01:59:17.274358 9604 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43338
I20240502 01:59:17.274895 6615 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240502 01:59:17.275707 9820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:59:17.275909 6615 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240502 01:59:17.275969 6615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714615076189642-6615-0/minicluster-data/ts-2-root
uuid: "6b35082b827b4212a4f4098dac93a5a6"
format_stamp: "Formatted at 2024-05-02 01:59:17 on dist-test-slave-89cm"
I20240502 01:59:17.276032 6615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714615076189642-6615-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714615076189642-6615-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714615076189642-6615-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:59:17.291929 6615 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:59:17.292366 6615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:59:17.292656 6615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:59:17.293148 6615 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:59:17.293190 6615 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:59:17.293234 6615 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:59:17.293267 6615 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:59:17.297536 6615 rpc_server.cc:307] RPC server started. Bound to: 127.6.117.195:33405
I20240502 01:59:17.297593 9883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.117.195:33405 every 8 connection(s)
I20240502 01:59:17.301810 9884 heartbeater.cc:347] Connected to a master server at 127.6.117.254:33303
I20240502 01:59:17.301918 9884 heartbeater.cc:464] Registering TS with master...
I20240502 01:59:17.302150 9884 heartbeater.cc:511] Master 127.6.117.254:33303 requested a full tablet report, sending...
I20240502 01:59:17.302793 9604 ts_manager.cc:194] Registered new tserver with Master: 6b35082b827b4212a4f4098dac93a5a6 (127.6.117.195:33405)
I20240502 01:59:17.303324 9604 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43354
I20240502 01:59:17.303661 6615 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.005730541s
I20240502 01:59:18.249316 9733 heartbeater.cc:503] Master 127.6.117.254:33303 was elected leader, sending a full tablet report...
I20240502 01:59:18.275266 9809 heartbeater.cc:503] Master 127.6.117.254:33303 was elected leader, sending a full tablet report...
I20240502 01:59:18.304203 9884 heartbeater.cc:503] Master 127.6.117.254:33303 was elected leader, sending a full tablet report...
I20240502 01:59:18.334537 6615 test_util.cc:268] Using random seed: -210738106
I20240502 01:59:18.342208 9604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43372:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 01:59:18.342643 9604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 01:59:18.352209 9848 tablet_service.cc:1467] Processing CreateTablet for tablet 3d24a516ab7d4be094879acb9d68eb83 (DEFAULT_TABLE table=test-workload [id=b5b9e7aefc2d4a299eb9834653702e6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:18.352150 9769 tablet_service.cc:1467] Processing CreateTablet for tablet 3d24a516ab7d4be094879acb9d68eb83 (DEFAULT_TABLE table=test-workload [id=b5b9e7aefc2d4a299eb9834653702e6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:18.352540 9769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d24a516ab7d4be094879acb9d68eb83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:18.352540 9848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d24a516ab7d4be094879acb9d68eb83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:18.353150 9697 tablet_service.cc:1467] Processing CreateTablet for tablet 3d24a516ab7d4be094879acb9d68eb83 (DEFAULT_TABLE table=test-workload [id=b5b9e7aefc2d4a299eb9834653702e6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:18.353407 9697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d24a516ab7d4be094879acb9d68eb83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:18.355298 9907 tablet_bootstrap.cc:492] T 3d24a516ab7d4be094879acb9d68eb83 P 86b8193cab1b4138a57d00fc6f8927b1: Bootstrap starting.
I20240502 01:59:18.355329 9906 tablet_bootstrap.cc:492] T 3d24a516ab7d4be094879acb9d68eb83 P c29ceaea1f7c4cc89ade5b8c12952856: Bootstrap starting.
I20240502 01:59:18.356333 9906 tablet_bootstrap.cc:654] T 3d24a516ab7d4be094879acb9d68eb83 P c29ceaea1f7c4cc89ade5b8c12952856: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:18.356333 9907 tablet_bootstrap.cc:654] T 3d24a516ab7d4be094879acb9d68eb83 P 86b8193cab1b4138a57d00fc6f8927b1: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:18.356423 9905 tablet_bootstrap.cc:492] T 3d24a516ab7d4be094879acb9d68eb83 P 6b35082b827b4212a4f4098dac93a5a6: Bootstrap starting.
I20240502 01:59:18.357210 9905 tablet_bootstrap.cc:654] T 3d24a516ab7d4be094879acb9d68eb83 P 6b35082b827b4212a4f4098dac93a5a6: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:18.357523 9906 tablet_bootstrap.cc:492] T 3d24a516ab7d4be094879acb9d68eb83 P c29ceaea1f7c4cc89ade5b8c12952856: No bootstrap required, opened a new log
I20240502 01:59:18.357523 9907 tablet_bootstrap.cc:492] T 3d24a516ab7d4be094879acb9d68eb83 P 86b8193cab1b4138a57d00fc6f8927b1: No bootstrap required, opened a new log
I20240502 01:59:18.357647 9906 ts_tablet_manager.cc:1367] T 3d24a516ab7d4be094879acb9d68eb83 P c29ceaea1f7c4cc89ade5b8c12952856: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:59:18.357647 9907 ts_tablet_manager.cc:1367] T 3d24a516ab7d4be094879acb9d68eb83 P 86b8193cab1b4138a57d00fc6f8927b1: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:59:18.358099 9907 raft_consensus.cc:348] T 3d24a516ab7d4be094879acb9d68eb83 P 86b8193cab1b4138a57d00fc6f8927b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b35082b827b4212a4f4098dac93a5a6" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 33405 } } peers { permanent_uuid: "c29ceaea1f7c4cc89ade5b8c12952856" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 34609 } } peers { permanent_uuid: "86b8193cab1b4138a57d00fc6f8927b1" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 40583 } }
I20240502 01:59:18.358197 9905 tablet_bootstrap.cc:492] T 3d24a516ab7d4be094879acb9d68eb83 P 6b35082b827b4212a4f4098dac93a5a6: No bootstrap required, opened a new log
I20240502 01:59:18.358142 9906 raft_consensus.cc:348] T 3d24a516ab7d4be094879acb9d68eb83 P c29ceaea1f7c4cc89ade5b8c12952856 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b35082b827b4212a4f4098dac93a5a6" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 33405 } } peers { permanent_uuid: "c29ceaea1f7c4cc89ade5b8c12952856" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 34609 } } peers { permanent_uuid: "86b8193cab1b4138a57d00fc6f8927b1" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 40583 } }
I20240502 01:59:18.358209 9907 raft_consensus.cc:374] T 3d24a516ab7d4be094879acb9d68eb83 P 86b8193cab1b4138a57d00fc6f8927b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:18.358237 9906 raft_consensus.cc:374] T 3d24a516ab7d4be094879acb9d68eb83 P c29ceaea1f7c4cc89ade5b8c12952856 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:18.358265 9905 ts_tablet_manager.cc:1367] T 3d24a516ab7d4be094879acb9d68eb83 P 6b35082b827b4212a4f4098dac93a5a6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:59:18.358282 9906 raft_consensus.cc:729] T 3d24a516ab7d4be094879acb9d68eb83 P c29ceaea1f7c4cc89ade5b8c12952856 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c29ceaea1f7c4cc89ade5b8c12952856, State: Initialized, Role: FOLLOWER
I20240502 01:59:18.358245 9907 raft_consensus.cc:729] T 3d24a516ab7d4be094879acb9d68eb83 P 86b8193cab1b4138a57d00fc6f8927b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86b8193cab1b4138a57d00fc6f8927b1, State: Initialized, Role: FOLLOWER
I20240502 01:59:18.358428 9907 consensus_queue.cc:260] T 3d24a516ab7d4be094879acb9d68eb83 P 86b8193cab1b4138a57d00fc6f8927b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b35082b827b4212a4f4098dac93a5a6" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 33405 } } peers { permanent_uuid: "c29ceaea1f7c4cc89ade5b8c12952856" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 34609 } } peers { permanent_uuid: "86b8193cab1b4138a57d00fc6f8927b1" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 40583 } }
I20240502 01:59:18.358428 9906 consensus_queue.cc:260] T 3d24a516ab7d4be094879acb9d68eb83 P c29ceaea1f7c4cc89ade5b8c12952856 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b35082b827b4212a4f4098dac93a5a6" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 33405 } } peers { permanent_uuid: "c29ceaea1f7c4cc89ade5b8c12952856" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 34609 } } peers { permanent_uuid: "86b8193cab1b4138a57d00fc6f8927b1" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 40583 } }
I20240502 01:59:18.358793 9907 ts_tablet_manager.cc:1398] T 3d24a516ab7d4be094879acb9d68eb83 P 86b8193cab1b4138a57d00fc6f8927b1: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:59:18.358803 9906 ts_tablet_manager.cc:1398] T 3d24a516ab7d4be094879acb9d68eb83 P c29ceaea1f7c4cc89ade5b8c12952856: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:59:18.359190 9905 raft_consensus.cc:348] T 3d24a516ab7d4be094879acb9d68eb83 P 6b35082b827b4212a4f4098dac93a5a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b35082b827b4212a4f4098dac93a5a6" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 33405 } } peers { permanent_uuid: "c29ceaea1f7c4cc89ade5b8c12952856" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 34609 } } peers { permanent_uuid: "86b8193cab1b4138a57d00fc6f8927b1" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 40583 } }
I20240502 01:59:18.359277 9905 raft_consensus.cc:374] T 3d24a516ab7d4be094879acb9d68eb83 P 6b35082b827b4212a4f4098dac93a5a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:18.359304 9905 raft_consensus.cc:729] T 3d24a516ab7d4be094879acb9d68eb83 P 6b35082b827b4212a4f4098dac93a5a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b35082b827b4212a4f4098dac93a5a6, State: Initialized, Role: FOLLOWER
I20240502 01:59:18.359409 9905 consensus_queue.cc:260] T 3d24a516ab7d4be094879acb9d68eb83 P 6b35082b827b4212a4f4098dac93a5a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b35082b827b4212a4f4098dac93a5a6" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 33405 } } peers { permanent_uuid: "c29ceaea1f7c4cc89ade5b8c12952856" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 34609 } } peers { permanent_uuid: "86b8193cab1b4138a57d00fc6f8927b1" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 40583 } }
I20240502 01:59:18.359736 9905 ts_tablet_manager.cc:1398] T 3d24a516ab7d4be094879acb9d68eb83 P 6b35082b827b4212a4f4098dac93a5a6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:18.649076 9911 raft_consensus.cc:482] T 3d24a516ab7d4be094879acb9d68eb83 P 86b8193cab1b4138a57d00fc6f8927b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:18.649238 9911 raft_consensus.cc:504] T 3d24a516ab7d4be094879acb9d68eb83 P 86b8193cab1b4138a57d00fc6f8927b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b35082b827b4212a4f4098dac93a5a6" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 33405 } } peers { permanent_uuid: "c29ceaea1f7c4cc89ade5b8c12952856" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 34609 } } peers { permanent_uuid: "86b8193cab1b4138a57d00fc6f8927b1" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 40583 } }
I20240502 01:59:18.649752 9911 leader_election.cc:288] T 3d24a516ab7d4be094879acb9d68eb83 P 86b8193cab1b4138a57d00fc6f8927b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b35082b827b4212a4f4098dac93a5a6 (127.6.117.195:33405), c29ceaea1f7c4cc89ade5b8c12952856 (127.6.117.194:34609)
I20240502 01:59:18.654443 9858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d24a516ab7d4be094879acb9d68eb83" candidate_uuid: "86b8193cab1b4138a57d00fc6f8927b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b35082b827b4212a4f4098dac93a5a6" is_pre_election: true
I20240502 01:59:18.654443 9783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d24a516ab7d4be094879acb9d68eb83" candidate_uuid: "86b8193cab1b4138a57d00fc6f8927b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c29ceaea1f7c4cc89ade5b8c12952856" is_pre_election: true
I20240502 01:59:18.654675 9783 raft_consensus.cc:2454] T 3d24a516ab7d4be094879acb9d68eb83 P c29ceaea1f7c4cc89ade5b8c12952856 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86b8193cab1b4138a57d00fc6f8927b1 in term 0.
I20240502 01:59:18.654675 9858 raft_consensus.cc:2454] T 3d24a516ab7d4be094879acb9d68eb83 P 6b35082b827b4212a4f4098dac93a5a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86b8193cab1b4138a57d00fc6f8927b1 in term 0.
I20240502 01:59:18.655035 9673 leader_election.cc:302] T 3d24a516ab7d4be094879acb9d68eb83 P 86b8193cab1b4138a57d00fc6f8927b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86b8193cab1b4138a57d00fc6f8927b1, c29ceaea1f7c4cc89ade5b8c12952856; no voters:
I20240502 01:59:18.655233 9911 raft_consensus.cc:2781] T 3d24a516ab7d4be094879acb9d68eb83 P 86b8193cab1b4138a57d00fc6f8927b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:18.655304 9911 raft_consensus.cc:482] T 3d24a516ab7d4be094879acb9d68eb83 P 86b8193cab1b4138a57d00fc6f8927b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:18.655331 9911 raft_consensus.cc:3037] T 3d24a516ab7d4be094879acb9d68eb83 P 86b8193cab1b4138a57d00fc6f8927b1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:18.656183 9911 raft_consensus.cc:504] T 3d24a516ab7d4be094879acb9d68eb83 P 86b8193cab1b4138a57d00fc6f8927b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b35082b827b4212a4f4098dac93a5a6" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 33405 } } peers { permanent_uuid: "c29ceaea1f7c4cc89ade5b8c12952856" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 34609 } } peers { permanent_uuid: "86b8193cab1b4138a57d00fc6f8927b1" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 40583 } }
I20240502 01:59:18.656560 9911 leader_election.cc:288] T 3d24a516ab7d4be094879acb9d68eb83 P 86b8193cab1b4138a57d00fc6f8927b1 [CANDIDATE]: Term 1 election: Requested vote from peers 6b35082b827b4212a4f4098dac93a5a6 (127.6.117.195:33405), c29ceaea1f7c4cc89ade5b8c12952856 (127.6.117.194:34609)
I20240502 01:59:18.656879 9783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d24a516ab7d4be094879acb9d68eb83" candidate_uuid: "86b8193cab1b4138a57d00fc6f8927b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c29ceaea1f7c4cc89ade5b8c12952856"
I20240502 01:59:18.656981 9858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d24a516ab7d4be094879acb9d68eb83" candidate_uuid: "86b8193cab1b4138a57d00fc6f8927b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b35082b827b4212a4f4098dac93a5a6"
I20240502 01:59:18.657019 9783 raft_consensus.cc:3037] T 3d24a516ab7d4be094879acb9d68eb83 P c29ceaea1f7c4cc89ade5b8c12952856 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:18.657066 9858 raft_consensus.cc:3037] T 3d24a516ab7d4be094879acb9d68eb83 P 6b35082b827b4212a4f4098dac93a5a6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:18.657979 9858 raft_consensus.cc:2454] T 3d24a516ab7d4be094879acb9d68eb83 P 6b35082b827b4212a4f4098dac93a5a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86b8193cab1b4138a57d00fc6f8927b1 in term 1.
I20240502 01:59:18.657979 9783 raft_consensus.cc:2454] T 3d24a516ab7d4be094879acb9d68eb83 P c29ceaea1f7c4cc89ade5b8c12952856 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86b8193cab1b4138a57d00fc6f8927b1 in term 1.
I20240502 01:59:18.658293 9670 leader_election.cc:302] T 3d24a516ab7d4be094879acb9d68eb83 P 86b8193cab1b4138a57d00fc6f8927b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b35082b827b4212a4f4098dac93a5a6, 86b8193cab1b4138a57d00fc6f8927b1; no voters:
I20240502 01:59:18.658465 9911 raft_consensus.cc:2781] T 3d24a516ab7d4be094879acb9d68eb83 P 86b8193cab1b4138a57d00fc6f8927b1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:18.658674 9911 raft_consensus.cc:686] T 3d24a516ab7d4be094879acb9d68eb83 P 86b8193cab1b4138a57d00fc6f8927b1 [term 1 LEADER]: Becoming Leader. State: Replica: 86b8193cab1b4138a57d00fc6f8927b1, State: Running, Role: LEADER
I20240502 01:59:18.658825 9911 consensus_queue.cc:237] T 3d24a516ab7d4be094879acb9d68eb83 P 86b8193cab1b4138a57d00fc6f8927b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b35082b827b4212a4f4098dac93a5a6" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 33405 } } peers { permanent_uuid: "c29ceaea1f7c4cc89ade5b8c12952856" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 34609 } } peers { permanent_uuid: "86b8193cab1b4138a57d00fc6f8927b1" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 40583 } }
I20240502 01:59:18.660375 9603 catalog_manager.cc:5484] T 3d24a516ab7d4be094879acb9d68eb83 P 86b8193cab1b4138a57d00fc6f8927b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 86b8193cab1b4138a57d00fc6f8927b1 (127.6.117.193). New cstate: current_term: 1 leader_uuid: "86b8193cab1b4138a57d00fc6f8927b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b35082b827b4212a4f4098dac93a5a6" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 33405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c29ceaea1f7c4cc89ade5b8c12952856" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 34609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "86b8193cab1b4138a57d00fc6f8927b1" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 40583 } health_report { overall_health: HEALTHY } } }
I20240502 01:59:19.068219 9911 consensus_queue.cc:1035] T 3d24a516ab7d4be094879acb9d68eb83 P 86b8193cab1b4138a57d00fc6f8927b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b35082b827b4212a4f4098dac93a5a6" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 33405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:19.071399 9916 consensus_queue.cc:1035] T 3d24a516ab7d4be094879acb9d68eb83 P 86b8193cab1b4138a57d00fc6f8927b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c29ceaea1f7c4cc89ade5b8c12952856" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 34609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:19.145174 9658 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:59:19.145455 9652 placement_policy_util.cc:407] tablet 3d24a516ab7d4be094879acb9d68eb83: detected majority of replicas (2 of 3) at location L0
I20240502 01:59:19.145517 9658 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:59:19.145591 9658 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:59:20.147346 9652 placement_policy_util.cc:407] tablet 3d24a516ab7d4be094879acb9d68eb83: detected majority of replicas (2 of 3) at location L0
I20240502 01:59:21.145905 9658 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:59:21.146235 9658 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:59:21.146292 9658 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:59:21.148869 9652 placement_policy_util.cc:407] tablet 3d24a516ab7d4be094879acb9d68eb83: detected majority of replicas (2 of 3) at location L0
I20240502 01:59:22.150418 9652 placement_policy_util.cc:407] tablet 3d24a516ab7d4be094879acb9d68eb83: detected majority of replicas (2 of 3) at location L0
I20240502 01:59:23.146607 9658 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:59:23.146903 9658 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:59:23.146956 9658 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:59:23.151978 9652 placement_policy_util.cc:407] tablet 3d24a516ab7d4be094879acb9d68eb83: detected majority of replicas (2 of 3) at location L0
I20240502 01:59:24.153561 9652 placement_policy_util.cc:407] tablet 3d24a516ab7d4be094879acb9d68eb83: detected majority of replicas (2 of 3) at location L0
I20240502 01:59:25.147217 9658 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:59:25.147511 9658 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:59:25.147565 9658 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:59:25.155099 9652 placement_policy_util.cc:407] tablet 3d24a516ab7d4be094879acb9d68eb83: detected majority of replicas (2 of 3) at location L0
I20240502 01:59:26.156910 9652 placement_policy_util.cc:407] tablet 3d24a516ab7d4be094879acb9d68eb83: detected majority of replicas (2 of 3) at location L0
I20240502 01:59:27.147874 9658 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:59:27.148213 9658 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:59:27.148300 9658 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:59:27.158600 9652 placement_policy_util.cc:407] tablet 3d24a516ab7d4be094879acb9d68eb83: detected majority of replicas (2 of 3) at location L0
I20240502 01:59:28.160200 9652 placement_policy_util.cc:407] tablet 3d24a516ab7d4be094879acb9d68eb83: detected majority of replicas (2 of 3) at location L0
I20240502 01:59:29.148669 9658 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:59:29.148975 9658 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:59:29.149032 9658 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:59:29.161856 9652 placement_policy_util.cc:407] tablet 3d24a516ab7d4be094879acb9d68eb83: detected majority of replicas (2 of 3) at location L0
I20240502 01:59:29.762562 6615 tablet_server.cc:178] TabletServer@127.6.117.193:0 shutting down...
I20240502 01:59:29.765805 6615 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:59:29.766008 6615 tablet_replica.cc:330] T 3d24a516ab7d4be094879acb9d68eb83 P 86b8193cab1b4138a57d00fc6f8927b1: stopping tablet replica
I20240502 01:59:29.766125 6615 raft_consensus.cc:2229] T 3d24a516ab7d4be094879acb9d68eb83 P 86b8193cab1b4138a57d00fc6f8927b1 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:59:29.766346 6615 raft_consensus.cc:2258] T 3d24a516ab7d4be094879acb9d68eb83 P 86b8193cab1b4138a57d00fc6f8927b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:29.779281 6615 tablet_server.cc:194] TabletServer@127.6.117.193:0 shutdown complete.
I20240502 01:59:29.781445 6615 tablet_server.cc:178] TabletServer@127.6.117.194:0 shutting down...
I20240502 01:59:29.786316 6615 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:59:29.786540 6615 tablet_replica.cc:330] T 3d24a516ab7d4be094879acb9d68eb83 P c29ceaea1f7c4cc89ade5b8c12952856: stopping tablet replica
I20240502 01:59:29.786653 6615 raft_consensus.cc:2229] T 3d24a516ab7d4be094879acb9d68eb83 P c29ceaea1f7c4cc89ade5b8c12952856 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:29.786787 6615 raft_consensus.cc:2258] T 3d24a516ab7d4be094879acb9d68eb83 P c29ceaea1f7c4cc89ade5b8c12952856 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:29.799269 6615 tablet_server.cc:194] TabletServer@127.6.117.194:0 shutdown complete.
I20240502 01:59:29.801311 6615 tablet_server.cc:178] TabletServer@127.6.117.195:0 shutting down...
I20240502 01:59:29.804868 6615 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:59:29.805055 6615 tablet_replica.cc:330] T 3d24a516ab7d4be094879acb9d68eb83 P 6b35082b827b4212a4f4098dac93a5a6: stopping tablet replica
I20240502 01:59:29.805171 6615 raft_consensus.cc:2229] T 3d24a516ab7d4be094879acb9d68eb83 P 6b35082b827b4212a4f4098dac93a5a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:29.805282 6615 raft_consensus.cc:2258] T 3d24a516ab7d4be094879acb9d68eb83 P 6b35082b827b4212a4f4098dac93a5a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:29.817677 6615 tablet_server.cc:194] TabletServer@127.6.117.195:0 shutdown complete.
I20240502 01:59:29.819675 6615 master.cc:537] Master@127.6.117.254:33303 shutting down...
I20240502 01:59:29.822525 6615 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 78a29320f19d4bea835089f246fbeffa [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:59:29.822688 6615 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 78a29320f19d4bea835089f246fbeffa [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:29.822752 6615 tablet_replica.cc:330] T 00000000000000000000000000000000 P 78a29320f19d4bea835089f246fbeffa: stopping tablet replica
I20240502 01:59:29.824839 6615 master.cc:552] Master@127.6.117.254:33303 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy (12716 ms)
[ RUN ] AutoRebalancerTest.TestMaxMovesPerServer
I20240502 01:59:29.830883 6615 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.6.117.254:37527
I20240502 01:59:29.831194 6615 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:59:29.833101 9929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:59:29.833428 9932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:59:29.833720 9930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:59:29.834146 6615 server_base.cc:1034] running on GCE node
I20240502 01:59:29.834306 6615 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:59:29.834348 6615 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:59:29.834374 6615 hybrid_clock.cc:648] HybridClock initialized: now 1714615169834374 us; error 0 us; skew 500 ppm
I20240502 01:59:29.835358 6615 webserver.cc:460] Webserver started at http://127.6.117.254:42413/ using document root <none> and password file <none>
I20240502 01:59:29.835505 6615 fs_manager.cc:359] Metadata directory not provided
I20240502 01:59:29.835556 6615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:59:29.835633 6615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:59:29.836022 6615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714615076189642-6615-0/minicluster-data/master-0-root/instance:
uuid: "1a94d7d439cf40c1bea6fbd8a61822e2"
format_stamp: "Formatted at 2024-05-02 01:59:29 on dist-test-slave-89cm"
I20240502 01:59:29.837478 6615 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240502 01:59:29.838373 9937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:59:29.838608 6615 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:29.838672 6615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714615076189642-6615-0/minicluster-data/master-0-root
uuid: "1a94d7d439cf40c1bea6fbd8a61822e2"
format_stamp: "Formatted at 2024-05-02 01:59:29 on dist-test-slave-89cm"
I20240502 01:59:29.838747 6615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714615076189642-6615-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714615076189642-6615-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714615076189642-6615-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:59:29.877528 6615 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:59:29.877943 6615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:59:29.881973 6615 rpc_server.cc:307] RPC server started. Bound to: 127.6.117.254:37527
I20240502 01:59:29.883767 9989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.117.254:37527 every 8 connection(s)
I20240502 01:59:29.884344 9990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:29.886174 9990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a94d7d439cf40c1bea6fbd8a61822e2: Bootstrap starting.
I20240502 01:59:29.886972 9990 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a94d7d439cf40c1bea6fbd8a61822e2: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:29.888180 9990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a94d7d439cf40c1bea6fbd8a61822e2: No bootstrap required, opened a new log
I20240502 01:59:29.888622 9990 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a94d7d439cf40c1bea6fbd8a61822e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a94d7d439cf40c1bea6fbd8a61822e2" member_type: VOTER }
I20240502 01:59:29.888721 9990 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a94d7d439cf40c1bea6fbd8a61822e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:29.888742 9990 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a94d7d439cf40c1bea6fbd8a61822e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a94d7d439cf40c1bea6fbd8a61822e2, State: Initialized, Role: FOLLOWER
I20240502 01:59:29.888868 9990 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a94d7d439cf40c1bea6fbd8a61822e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a94d7d439cf40c1bea6fbd8a61822e2" member_type: VOTER }
I20240502 01:59:29.888947 9990 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a94d7d439cf40c1bea6fbd8a61822e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:59:29.888968 9990 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a94d7d439cf40c1bea6fbd8a61822e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:59:29.889001 9990 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a94d7d439cf40c1bea6fbd8a61822e2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:29.889753 9990 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a94d7d439cf40c1bea6fbd8a61822e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a94d7d439cf40c1bea6fbd8a61822e2" member_type: VOTER }
I20240502 01:59:29.889890 9990 leader_election.cc:302] T 00000000000000000000000000000000 P 1a94d7d439cf40c1bea6fbd8a61822e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a94d7d439cf40c1bea6fbd8a61822e2; no voters:
I20240502 01:59:29.890093 9990 leader_election.cc:288] T 00000000000000000000000000000000 P 1a94d7d439cf40c1bea6fbd8a61822e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:59:29.890241 9993 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a94d7d439cf40c1bea6fbd8a61822e2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:29.890425 9993 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a94d7d439cf40c1bea6fbd8a61822e2 [term 1 LEADER]: Becoming Leader. State: Replica: 1a94d7d439cf40c1bea6fbd8a61822e2, State: Running, Role: LEADER
I20240502 01:59:29.890560 9993 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a94d7d439cf40c1bea6fbd8a61822e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a94d7d439cf40c1bea6fbd8a61822e2" member_type: VOTER }
I20240502 01:59:29.890617 9990 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a94d7d439cf40c1bea6fbd8a61822e2 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 01:59:29.890998 9994 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a94d7d439cf40c1bea6fbd8a61822e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a94d7d439cf40c1bea6fbd8a61822e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a94d7d439cf40c1bea6fbd8a61822e2" member_type: VOTER } }
I20240502 01:59:29.891129 9994 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a94d7d439cf40c1bea6fbd8a61822e2 [sys.catalog]: This master's current role is: LEADER
I20240502 01:59:29.891023 9995 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a94d7d439cf40c1bea6fbd8a61822e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a94d7d439cf40c1bea6fbd8a61822e2. Latest consensus state: current_term: 1 leader_uuid: "1a94d7d439cf40c1bea6fbd8a61822e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a94d7d439cf40c1bea6fbd8a61822e2" member_type: VOTER } }
I20240502 01:59:29.891254 9995 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a94d7d439cf40c1bea6fbd8a61822e2 [sys.catalog]: This master's current role is: LEADER
I20240502 01:59:29.891745 9998 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 01:59:29.892576 9998 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 01:59:29.892746 6615 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 01:59:29.894459 9998 catalog_manager.cc:1347] Generated new cluster ID: f8649f845a1b40259426d34d1376d851
I20240502 01:59:29.894510 9998 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 01:59:29.908208 9998 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 01:59:29.908836 9998 catalog_manager.cc:1504] Loading token signing keys...
I20240502 01:59:29.917703 9998 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 1a94d7d439cf40c1bea6fbd8a61822e2: Generated new TSK 0
I20240502 01:59:29.917917 9998 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 01:59:29.925557 6615 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:59:29.927910 10014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:59:29.928243 10012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:59:29.928570 10011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:59:29.929008 6615 server_base.cc:1034] running on GCE node
I20240502 01:59:29.929173 6615 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:59:29.929209 6615 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:59:29.929225 6615 hybrid_clock.cc:648] HybridClock initialized: now 1714615169929224 us; error 0 us; skew 500 ppm
I20240502 01:59:29.930191 6615 webserver.cc:460] Webserver started at http://127.6.117.193:44591/ using document root <none> and password file <none>
I20240502 01:59:29.930359 6615 fs_manager.cc:359] Metadata directory not provided
I20240502 01:59:29.930406 6615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:59:29.930483 6615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:59:29.930873 6615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714615076189642-6615-0/minicluster-data/ts-0-root/instance:
uuid: "bdbecaeb8f214529b2012a9c003de2ff"
format_stamp: "Formatted at 2024-05-02 01:59:29 on dist-test-slave-89cm"
I20240502 01:59:29.932312 6615 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240502 01:59:29.933220 10019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:59:29.933442 6615 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:29.933511 6615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714615076189642-6615-0/minicluster-data/ts-0-root
uuid: "bdbecaeb8f214529b2012a9c003de2ff"
format_stamp: "Formatted at 2024-05-02 01:59:29 on dist-test-slave-89cm"
I20240502 01:59:29.933584 6615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714615076189642-6615-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714615076189642-6615-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714615076189642-6615-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:59:29.947054 6615 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:59:29.947463 6615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:59:29.947764 6615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:59:29.948243 6615 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:59:29.948356 6615 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:59:29.948406 6615 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:59:29.948433 6615 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:59:29.952695 6615 rpc_server.cc:307] RPC server started. Bound to: 127.6.117.193:34597
I20240502 01:59:29.952741 10082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.117.193:34597 every 8 connection(s)
I20240502 01:59:29.953505 6615 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:59:29.955619 10087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:59:29.956101 6615 server_base.cc:1034] running on GCE node
W20240502 01:59:29.956406 10088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:59:29.956688 10090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:59:29.956940 6615 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:59:29.956992 6615 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:59:29.957011 6615 hybrid_clock.cc:648] HybridClock initialized: now 1714615169957011 us; error 0 us; skew 500 ppm
I20240502 01:59:29.957885 6615 webserver.cc:460] Webserver started at http://127.6.117.194:42651/ using document root <none> and password file <none>
I20240502 01:59:29.958072 6615 fs_manager.cc:359] Metadata directory not provided
I20240502 01:59:29.958137 6615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:59:29.958214 6615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:59:29.958431 10083 heartbeater.cc:347] Connected to a master server at 127.6.117.254:37527
I20240502 01:59:29.958526 10083 heartbeater.cc:464] Registering TS with master...
I20240502 01:59:29.958722 10083 heartbeater.cc:511] Master 127.6.117.254:37527 requested a full tablet report, sending...
I20240502 01:59:29.958716 6615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714615076189642-6615-0/minicluster-data/ts-1-root/instance:
uuid: "4b7f7ba2ac084e829c4304c16a94894f"
format_stamp: "Formatted at 2024-05-02 01:59:29 on dist-test-slave-89cm"
I20240502 01:59:29.959435 9954 ts_manager.cc:194] Registered new tserver with Master: bdbecaeb8f214529b2012a9c003de2ff (127.6.117.193:34597)
I20240502 01:59:29.960197 9954 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43124
I20240502 01:59:29.960356 6615 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:29.961300 10095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:59:29.961576 6615 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:29.961643 6615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714615076189642-6615-0/minicluster-data/ts-1-root
uuid: "4b7f7ba2ac084e829c4304c16a94894f"
format_stamp: "Formatted at 2024-05-02 01:59:29 on dist-test-slave-89cm"
I20240502 01:59:29.961714 6615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714615076189642-6615-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714615076189642-6615-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714615076189642-6615-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:59:29.970129 6615 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:59:29.970544 6615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:59:29.970850 6615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:59:29.971321 6615 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:59:29.971362 6615 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:59:29.971405 6615 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:59:29.971431 6615 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:59:29.975822 6615 rpc_server.cc:307] RPC server started. Bound to: 127.6.117.194:46805
I20240502 01:59:29.975867 10158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.117.194:46805 every 8 connection(s)
I20240502 01:59:29.976686 6615 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:59:29.979043 10162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:59:29.979257 10163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:59:29.979521 10165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:59:29.980042 6615 server_base.cc:1034] running on GCE node
I20240502 01:59:29.980221 6615 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:59:29.980293 6615 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:59:29.980348 6615 hybrid_clock.cc:648] HybridClock initialized: now 1714615169980347 us; error 0 us; skew 500 ppm
I20240502 01:59:29.980907 10159 heartbeater.cc:347] Connected to a master server at 127.6.117.254:37527
I20240502 01:59:29.981016 10159 heartbeater.cc:464] Registering TS with master...
I20240502 01:59:29.981258 10159 heartbeater.cc:511] Master 127.6.117.254:37527 requested a full tablet report, sending...
I20240502 01:59:29.981361 6615 webserver.cc:460] Webserver started at http://127.6.117.195:35691/ using document root <none> and password file <none>
I20240502 01:59:29.981513 6615 fs_manager.cc:359] Metadata directory not provided
I20240502 01:59:29.981575 6615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:59:29.981653 6615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:59:29.982131 9954 ts_manager.cc:194] Registered new tserver with Master: 4b7f7ba2ac084e829c4304c16a94894f (127.6.117.194:46805)
I20240502 01:59:29.982333 6615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714615076189642-6615-0/minicluster-data/ts-2-root/instance:
uuid: "e77d1c0d41034d1eb5a036dc17c071f5"
format_stamp: "Formatted at 2024-05-02 01:59:29 on dist-test-slave-89cm"
I20240502 01:59:29.982862 9954 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43140
I20240502 01:59:29.984071 6615 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240502 01:59:29.985010 10170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:59:29.985276 6615 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:29.985347 6615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714615076189642-6615-0/minicluster-data/ts-2-root
uuid: "e77d1c0d41034d1eb5a036dc17c071f5"
format_stamp: "Formatted at 2024-05-02 01:59:29 on dist-test-slave-89cm"
I20240502 01:59:29.985420 6615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714615076189642-6615-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714615076189642-6615-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714615076189642-6615-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:59:29.998478 6615 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:59:29.998950 6615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:59:29.999250 6615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:59:29.999732 6615 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:59:29.999773 6615 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:59:29.999806 6615 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:59:29.999820 6615 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:59:30.004175 6615 rpc_server.cc:307] RPC server started. Bound to: 127.6.117.195:43977
I20240502 01:59:30.004233 10233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.117.195:43977 every 8 connection(s)
I20240502 01:59:30.008494 10234 heartbeater.cc:347] Connected to a master server at 127.6.117.254:37527
I20240502 01:59:30.008606 10234 heartbeater.cc:464] Registering TS with master...
I20240502 01:59:30.008833 10234 heartbeater.cc:511] Master 127.6.117.254:37527 requested a full tablet report, sending...
I20240502 01:59:30.009428 9954 ts_manager.cc:194] Registered new tserver with Master: e77d1c0d41034d1eb5a036dc17c071f5 (127.6.117.195:43977)
I20240502 01:59:30.009948 9954 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43148
I20240502 01:59:30.010237 6615 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.005641694s
I20240502 01:59:30.961194 10083 heartbeater.cc:503] Master 127.6.117.254:37527 was elected leader, sending a full tablet report...
I20240502 01:59:30.983918 10159 heartbeater.cc:503] Master 127.6.117.254:37527 was elected leader, sending a full tablet report...
I20240502 01:59:31.010838 10234 heartbeater.cc:503] Master 127.6.117.254:37527 was elected leader, sending a full tablet report...
I20240502 01:59:31.036818 6615 test_util.cc:268] Using random seed: -198035829
I20240502 01:59:31.051095 9954 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43162:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\252\252\252\n\004\001\000TUU\025\004\001\000\376\377\377\037\004\001\000\250\252\252*\004\001\000RUU5\004\001\000\374\377\377?\004\001\000\246\252\252J\004\001\000PUUU\004\001\000\372\377\377_\004\001\000\244\252\252j\004\001\000NUUu""\004\001\000\252\252\252\n\004\001\000TUU\025\004\001\000\376\377\377\037\004\001\000\250\252\252*\004\001\000RUU5\004\001\000\374\377\377?\004\001\000\246\252\252J\004\001\000PUUU\004\001\000\372\377\377_\004\001\000\244\252\252j\004\001\000NUUu"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 01:59:31.051703 9954 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 01:59:31.066082 10047 tablet_service.cc:1467] Processing CreateTablet for tablet 4885222f238c4d8ea20ebc4c7e171694 (DEFAULT_TABLE table=test-workload [id=10a74377088c4a9890835737184010db]), partition=RANGE (key) PARTITION VALUES < 178956970
I20240502 01:59:31.066323 10044 tablet_service.cc:1467] Processing CreateTablet for tablet 73ea0bfb40ad43a68a6289ff1b35d005 (DEFAULT_TABLE table=test-workload [id=10a74377088c4a9890835737184010db]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20240502 01:59:31.066478 10047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4885222f238c4d8ea20ebc4c7e171694. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:31.066715 10043 tablet_service.cc:1467] Processing CreateTablet for tablet e7a8fc5afcab4b1db4ebf245b66df4be (DEFAULT_TABLE table=test-workload [id=10a74377088c4a9890835737184010db]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20240502 01:59:31.066792 10123 tablet_service.cc:1467] Processing CreateTablet for tablet 4885222f238c4d8ea20ebc4c7e171694 (DEFAULT_TABLE table=test-workload [id=10a74377088c4a9890835737184010db]), partition=RANGE (key) PARTITION VALUES < 178956970
I20240502 01:59:31.066946 10043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7a8fc5afcab4b1db4ebf245b66df4be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:31.067106 10123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4885222f238c4d8ea20ebc4c7e171694. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:31.068518 10042 tablet_service.cc:1467] Processing CreateTablet for tablet 66ccad5a808b42378ccecc9da81cee74 (DEFAULT_TABLE table=test-workload [id=10a74377088c4a9890835737184010db]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20240502 01:59:31.068768 10042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66ccad5a808b42378ccecc9da81cee74. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:31.068866 10046 tablet_service.cc:1467] Processing CreateTablet for tablet 1851ea6467584b0ea32a1f68696803d0 (DEFAULT_TABLE table=test-workload [id=10a74377088c4a9890835737184010db]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20240502 01:59:31.069101 10046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1851ea6467584b0ea32a1f68696803d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:31.069828 10123 tablet_service.cc:1467] Processing CreateTablet for tablet abf7d8b411bb46b68ac9ae77d74f8e2b (DEFAULT_TABLE table=test-workload [id=10a74377088c4a9890835737184010db]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20240502 01:59:31.069890 10116 tablet_service.cc:1467] Processing CreateTablet for tablet bceb844e50b84e278c0c49f4f4912e00 (DEFAULT_TABLE table=test-workload [id=10a74377088c4a9890835737184010db]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20240502 01:59:31.070067 10123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abf7d8b411bb46b68ac9ae77d74f8e2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:31.070310 10117 tablet_service.cc:1467] Processing CreateTablet for tablet b4409766ce994f698c31aede604a6927 (DEFAULT_TABLE table=test-workload [id=10a74377088c4a9890835737184010db]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20240502 01:59:31.070545 10117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4409766ce994f698c31aede604a6927. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:31.070894 10041 tablet_service.cc:1467] Processing CreateTablet for tablet b4409766ce994f698c31aede604a6927 (DEFAULT_TABLE table=test-workload [id=10a74377088c4a9890835737184010db]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20240502 01:59:31.070976 10045 tablet_service.cc:1467] Processing CreateTablet for tablet 580db11a61cb46f48d5f7507cfbdbb52 (DEFAULT_TABLE table=test-workload [id=10a74377088c4a9890835737184010db]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20240502 01:59:31.071130 10041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4409766ce994f698c31aede604a6927. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:31.071868 10045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 580db11a61cb46f48d5f7507cfbdbb52. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:31.072125 10115 tablet_service.cc:1467] Processing CreateTablet for tablet 39f1842ff1494d78bfa0811a799c4bb8 (DEFAULT_TABLE table=test-workload [id=10a74377088c4a9890835737184010db]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20240502 01:59:31.072417 10115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39f1842ff1494d78bfa0811a799c4bb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:31.072739 10117 tablet_service.cc:1467] Processing CreateTablet for tablet 13d726dd71b14e84bc061f537f81e91d (DEFAULT_TABLE table=test-workload [id=10a74377088c4a9890835737184010db]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20240502 01:59:31.072955 10117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13d726dd71b14e84bc061f537f81e91d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:31.073033 10044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73ea0bfb40ad43a68a6289ff1b35d005. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:31.074535 10254 tablet_bootstrap.cc:492] T 4885222f238c4d8ea20ebc4c7e171694 P 4b7f7ba2ac084e829c4304c16a94894f: Bootstrap starting.
I20240502 01:59:31.074506 10118 tablet_service.cc:1467] Processing CreateTablet for tablet 66ccad5a808b42378ccecc9da81cee74 (DEFAULT_TABLE table=test-workload [id=10a74377088c4a9890835737184010db]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20240502 01:59:31.074838 10118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66ccad5a808b42378ccecc9da81cee74. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:31.075064 10121 tablet_service.cc:1467] Processing CreateTablet for tablet 580db11a61cb46f48d5f7507cfbdbb52 (DEFAULT_TABLE table=test-workload [id=10a74377088c4a9890835737184010db]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20240502 01:59:31.075299 10121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 580db11a61cb46f48d5f7507cfbdbb52. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:31.075484 10047 tablet_service.cc:1467] Processing CreateTablet for tablet bceb844e50b84e278c0c49f4f4912e00 (DEFAULT_TABLE table=test-workload [id=10a74377088c4a9890835737184010db]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20240502 01:59:31.075659 10254 tablet_bootstrap.cc:654] T 4885222f238c4d8ea20ebc4c7e171694 P 4b7f7ba2ac084e829c4304c16a94894f: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:31.075685 10047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bceb844e50b84e278c0c49f4f4912e00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:31.076501 10046 tablet_service.cc:1467] Processing CreateTablet for tablet abf7d8b411bb46b68ac9ae77d74f8e2b (DEFAULT_TABLE table=test-workload [id=10a74377088c4a9890835737184010db]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20240502 01:59:31.076711 10046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abf7d8b411bb46b68ac9ae77d74f8e2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:31.074847 10122 tablet_service.cc:1467] Processing CreateTablet for tablet 1851ea6467584b0ea32a1f68696803d0 (DEFAULT_TABLE table=test-workload [id=10a74377088c4a9890835737184010db]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20240502 01:59:31.077268 10122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1851ea6467584b0ea32a1f68696803d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:31.077379 10120 tablet_service.cc:1467] Processing CreateTablet for tablet 73ea0bfb40ad43a68a6289ff1b35d005 (DEFAULT_TABLE table=test-workload [id=10a74377088c4a9890835737184010db]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20240502 01:59:31.077602 10120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73ea0bfb40ad43a68a6289ff1b35d005. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:31.078624 10045 tablet_service.cc:1467] Processing CreateTablet for tablet 39f1842ff1494d78bfa0811a799c4bb8 (DEFAULT_TABLE table=test-workload [id=10a74377088c4a9890835737184010db]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20240502 01:59:31.078842 10045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39f1842ff1494d78bfa0811a799c4bb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:31.079609 10119 tablet_service.cc:1467] Processing CreateTablet for tablet e7a8fc5afcab4b1db4ebf245b66df4be (DEFAULT_TABLE table=test-workload [id=10a74377088c4a9890835737184010db]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20240502 01:59:31.079901 10119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7a8fc5afcab4b1db4ebf245b66df4be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:31.079955 10198 tablet_service.cc:1467] Processing CreateTablet for tablet 4885222f238c4d8ea20ebc4c7e171694 (DEFAULT_TABLE table=test-workload [id=10a74377088c4a9890835737184010db]), partition=RANGE (key) PARTITION VALUES < 178956970
I20240502 01:59:31.080194 10198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4885222f238c4d8ea20ebc4c7e171694. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:31.074304 10114 tablet_service.cc:1467] Processing CreateTablet for tablet 19ff9c216a094fde86ea9f9648bf08cb (DEFAULT_TABLE table=test-workload [id=10a74377088c4a9890835737184010db]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20240502 01:59:31.080471 10114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19ff9c216a094fde86ea9f9648bf08cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:31.080483 10190 tablet_service.cc:1467] Processing CreateTablet for tablet 39f1842ff1494d78bfa0811a799c4bb8 (DEFAULT_TABLE table=test-workload [id=10a74377088c4a9890835737184010db]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20240502 01:59:31.081341 10190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39f1842ff1494d78bfa0811a799c4bb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:31.082011 10116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bceb844e50b84e278c0c49f4f4912e00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:31.082381 10198 tablet_service.cc:1467] Processing CreateTablet for tablet abf7d8b411bb46b68ac9ae77d74f8e2b (DEFAULT_TABLE table=test-workload [id=10a74377088c4a9890835737184010db]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20240502 01:59:31.082602 10198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abf7d8b411bb46b68ac9ae77d74f8e2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:31.083274 10190 tablet_service.cc:1467] Processing CreateTablet for tablet 13d726dd71b14e84bc061f537f81e91d (DEFAULT_TABLE table=test-workload [id=10a74377088c4a9890835737184010db]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20240502 01:59:31.083473 10190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13d726dd71b14e84bc061f537f81e91d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:31.084179 10041 tablet_service.cc:1467] Processing CreateTablet for tablet 19ff9c216a094fde86ea9f9648bf08cb (DEFAULT_TABLE table=test-workload [id=10a74377088c4a9890835737184010db]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20240502 01:59:31.084244 10042 tablet_service.cc:1467] Processing CreateTablet for tablet 13d726dd71b14e84bc061f537f81e91d (DEFAULT_TABLE table=test-workload [id=10a74377088c4a9890835737184010db]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20240502 01:59:31.084388 10197 tablet_service.cc:1467] Processing CreateTablet for tablet 1851ea6467584b0ea32a1f68696803d0 (DEFAULT_TABLE table=test-workload [id=10a74377088c4a9890835737184010db]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20240502 01:59:31.084470 10041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19ff9c216a094fde86ea9f9648bf08cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:31.084564 10042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13d726dd71b14e84bc061f537f81e91d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:31.084618 10197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1851ea6467584b0ea32a1f68696803d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:31.086277 10196 tablet_service.cc:1467] Processing CreateTablet for tablet 580db11a61cb46f48d5f7507cfbdbb52 (DEFAULT_TABLE table=test-workload [id=10a74377088c4a9890835737184010db]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20240502 01:59:31.086429 10255 tablet_bootstrap.cc:492] T 66ccad5a808b42378ccecc9da81cee74 P bdbecaeb8f214529b2012a9c003de2ff: Bootstrap starting.
I20240502 01:59:31.086510 10196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 580db11a61cb46f48d5f7507cfbdbb52. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:31.087002 10254 tablet_bootstrap.cc:492] T 4885222f238c4d8ea20ebc4c7e171694 P 4b7f7ba2ac084e829c4304c16a94894f: No bootstrap required, opened a new log
I20240502 01:59:31.087098 10254 ts_tablet_manager.cc:1367] T 4885222f238c4d8ea20ebc4c7e171694 P 4b7f7ba2ac084e829c4304c16a94894f: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240502 01:59:31.087154 10189 tablet_service.cc:1467] Processing CreateTablet for tablet 19ff9c216a094fde86ea9f9648bf08cb (DEFAULT_TABLE table=test-workload [id=10a74377088c4a9890835737184010db]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20240502 01:59:31.087417 10189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19ff9c216a094fde86ea9f9648bf08cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:31.087581 10254 raft_consensus.cc:348] T 4885222f238c4d8ea20ebc4c7e171694 P 4b7f7ba2ac084e829c4304c16a94894f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.087683 10254 raft_consensus.cc:374] T 4885222f238c4d8ea20ebc4c7e171694 P 4b7f7ba2ac084e829c4304c16a94894f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:31.087711 10254 raft_consensus.cc:729] T 4885222f238c4d8ea20ebc4c7e171694 P 4b7f7ba2ac084e829c4304c16a94894f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b7f7ba2ac084e829c4304c16a94894f, State: Initialized, Role: FOLLOWER
I20240502 01:59:31.087767 10255 tablet_bootstrap.cc:654] T 66ccad5a808b42378ccecc9da81cee74 P bdbecaeb8f214529b2012a9c003de2ff: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:31.087838 10254 consensus_queue.cc:260] T 4885222f238c4d8ea20ebc4c7e171694 P 4b7f7ba2ac084e829c4304c16a94894f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.088241 10254 ts_tablet_manager.cc:1398] T 4885222f238c4d8ea20ebc4c7e171694 P 4b7f7ba2ac084e829c4304c16a94894f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:31.088932 10254 tablet_bootstrap.cc:492] T abf7d8b411bb46b68ac9ae77d74f8e2b P 4b7f7ba2ac084e829c4304c16a94894f: Bootstrap starting.
I20240502 01:59:31.089243 10195 tablet_service.cc:1467] Processing CreateTablet for tablet 73ea0bfb40ad43a68a6289ff1b35d005 (DEFAULT_TABLE table=test-workload [id=10a74377088c4a9890835737184010db]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20240502 01:59:31.089505 10195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73ea0bfb40ad43a68a6289ff1b35d005. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:31.089587 10191 tablet_service.cc:1467] Processing CreateTablet for tablet bceb844e50b84e278c0c49f4f4912e00 (DEFAULT_TABLE table=test-workload [id=10a74377088c4a9890835737184010db]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20240502 01:59:31.089834 10191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bceb844e50b84e278c0c49f4f4912e00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:31.090286 10254 tablet_bootstrap.cc:654] T abf7d8b411bb46b68ac9ae77d74f8e2b P 4b7f7ba2ac084e829c4304c16a94894f: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:31.091454 10254 tablet_bootstrap.cc:492] T abf7d8b411bb46b68ac9ae77d74f8e2b P 4b7f7ba2ac084e829c4304c16a94894f: No bootstrap required, opened a new log
I20240502 01:59:31.091532 10254 ts_tablet_manager.cc:1367] T abf7d8b411bb46b68ac9ae77d74f8e2b P 4b7f7ba2ac084e829c4304c16a94894f: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 01:59:31.091800 10194 tablet_service.cc:1467] Processing CreateTablet for tablet e7a8fc5afcab4b1db4ebf245b66df4be (DEFAULT_TABLE table=test-workload [id=10a74377088c4a9890835737184010db]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20240502 01:59:31.091898 10192 tablet_service.cc:1467] Processing CreateTablet for tablet b4409766ce994f698c31aede604a6927 (DEFAULT_TABLE table=test-workload [id=10a74377088c4a9890835737184010db]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20240502 01:59:31.091935 10254 raft_consensus.cc:348] T abf7d8b411bb46b68ac9ae77d74f8e2b P 4b7f7ba2ac084e829c4304c16a94894f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.092020 10254 raft_consensus.cc:374] T abf7d8b411bb46b68ac9ae77d74f8e2b P 4b7f7ba2ac084e829c4304c16a94894f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:31.092039 10254 raft_consensus.cc:729] T abf7d8b411bb46b68ac9ae77d74f8e2b P 4b7f7ba2ac084e829c4304c16a94894f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b7f7ba2ac084e829c4304c16a94894f, State: Initialized, Role: FOLLOWER
I20240502 01:59:31.092056 10194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7a8fc5afcab4b1db4ebf245b66df4be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:31.092139 10254 consensus_queue.cc:260] T abf7d8b411bb46b68ac9ae77d74f8e2b P 4b7f7ba2ac084e829c4304c16a94894f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.092315 10193 tablet_service.cc:1467] Processing CreateTablet for tablet 66ccad5a808b42378ccecc9da81cee74 (DEFAULT_TABLE table=test-workload [id=10a74377088c4a9890835737184010db]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20240502 01:59:31.092418 10254 ts_tablet_manager.cc:1398] T abf7d8b411bb46b68ac9ae77d74f8e2b P 4b7f7ba2ac084e829c4304c16a94894f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:59:31.092587 10193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66ccad5a808b42378ccecc9da81cee74. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:31.092648 10254 tablet_bootstrap.cc:492] T b4409766ce994f698c31aede604a6927 P 4b7f7ba2ac084e829c4304c16a94894f: Bootstrap starting.
I20240502 01:59:31.093336 10254 tablet_bootstrap.cc:654] T b4409766ce994f698c31aede604a6927 P 4b7f7ba2ac084e829c4304c16a94894f: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:31.094408 10192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4409766ce994f698c31aede604a6927. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:31.095093 10254 tablet_bootstrap.cc:492] T b4409766ce994f698c31aede604a6927 P 4b7f7ba2ac084e829c4304c16a94894f: No bootstrap required, opened a new log
I20240502 01:59:31.095177 10254 ts_tablet_manager.cc:1367] T b4409766ce994f698c31aede604a6927 P 4b7f7ba2ac084e829c4304c16a94894f: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240502 01:59:31.095582 10254 raft_consensus.cc:348] T b4409766ce994f698c31aede604a6927 P 4b7f7ba2ac084e829c4304c16a94894f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.095762 10254 raft_consensus.cc:374] T b4409766ce994f698c31aede604a6927 P 4b7f7ba2ac084e829c4304c16a94894f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:31.096380 10254 raft_consensus.cc:729] T b4409766ce994f698c31aede604a6927 P 4b7f7ba2ac084e829c4304c16a94894f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b7f7ba2ac084e829c4304c16a94894f, State: Initialized, Role: FOLLOWER
I20240502 01:59:31.095860 10257 tablet_bootstrap.cc:492] T 4885222f238c4d8ea20ebc4c7e171694 P e77d1c0d41034d1eb5a036dc17c071f5: Bootstrap starting.
I20240502 01:59:31.096529 10254 consensus_queue.cc:260] T b4409766ce994f698c31aede604a6927 P 4b7f7ba2ac084e829c4304c16a94894f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.096846 10254 ts_tablet_manager.cc:1398] T b4409766ce994f698c31aede604a6927 P 4b7f7ba2ac084e829c4304c16a94894f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:59:31.096321 10255 tablet_bootstrap.cc:492] T 66ccad5a808b42378ccecc9da81cee74 P bdbecaeb8f214529b2012a9c003de2ff: No bootstrap required, opened a new log
I20240502 01:59:31.096997 10255 ts_tablet_manager.cc:1367] T 66ccad5a808b42378ccecc9da81cee74 P bdbecaeb8f214529b2012a9c003de2ff: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240502 01:59:31.097285 10254 tablet_bootstrap.cc:492] T 66ccad5a808b42378ccecc9da81cee74 P 4b7f7ba2ac084e829c4304c16a94894f: Bootstrap starting.
I20240502 01:59:31.097484 10257 tablet_bootstrap.cc:654] T 4885222f238c4d8ea20ebc4c7e171694 P e77d1c0d41034d1eb5a036dc17c071f5: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:31.097455 10255 raft_consensus.cc:348] T 66ccad5a808b42378ccecc9da81cee74 P bdbecaeb8f214529b2012a9c003de2ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.097692 10255 raft_consensus.cc:374] T 66ccad5a808b42378ccecc9da81cee74 P bdbecaeb8f214529b2012a9c003de2ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:31.097782 10255 raft_consensus.cc:729] T 66ccad5a808b42378ccecc9da81cee74 P bdbecaeb8f214529b2012a9c003de2ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdbecaeb8f214529b2012a9c003de2ff, State: Initialized, Role: FOLLOWER
I20240502 01:59:31.098007 10255 consensus_queue.cc:260] T 66ccad5a808b42378ccecc9da81cee74 P bdbecaeb8f214529b2012a9c003de2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.098464 10255 ts_tablet_manager.cc:1398] T 66ccad5a808b42378ccecc9da81cee74 P bdbecaeb8f214529b2012a9c003de2ff: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:31.098729 10257 tablet_bootstrap.cc:492] T 4885222f238c4d8ea20ebc4c7e171694 P e77d1c0d41034d1eb5a036dc17c071f5: No bootstrap required, opened a new log
I20240502 01:59:31.098847 10257 ts_tablet_manager.cc:1367] T 4885222f238c4d8ea20ebc4c7e171694 P e77d1c0d41034d1eb5a036dc17c071f5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:59:31.099000 10255 tablet_bootstrap.cc:492] T 1851ea6467584b0ea32a1f68696803d0 P bdbecaeb8f214529b2012a9c003de2ff: Bootstrap starting.
I20240502 01:59:31.099268 10254 tablet_bootstrap.cc:654] T 66ccad5a808b42378ccecc9da81cee74 P 4b7f7ba2ac084e829c4304c16a94894f: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:31.099303 10257 raft_consensus.cc:348] T 4885222f238c4d8ea20ebc4c7e171694 P e77d1c0d41034d1eb5a036dc17c071f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.099409 10257 raft_consensus.cc:374] T 4885222f238c4d8ea20ebc4c7e171694 P e77d1c0d41034d1eb5a036dc17c071f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:31.099443 10257 raft_consensus.cc:729] T 4885222f238c4d8ea20ebc4c7e171694 P e77d1c0d41034d1eb5a036dc17c071f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e77d1c0d41034d1eb5a036dc17c071f5, State: Initialized, Role: FOLLOWER
I20240502 01:59:31.099563 10257 consensus_queue.cc:260] T 4885222f238c4d8ea20ebc4c7e171694 P e77d1c0d41034d1eb5a036dc17c071f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.099925 10257 ts_tablet_manager.cc:1398] T 4885222f238c4d8ea20ebc4c7e171694 P e77d1c0d41034d1eb5a036dc17c071f5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:31.100117 10257 tablet_bootstrap.cc:492] T 39f1842ff1494d78bfa0811a799c4bb8 P e77d1c0d41034d1eb5a036dc17c071f5: Bootstrap starting.
I20240502 01:59:31.100131 10255 tablet_bootstrap.cc:654] T 1851ea6467584b0ea32a1f68696803d0 P bdbecaeb8f214529b2012a9c003de2ff: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:31.101156 10257 tablet_bootstrap.cc:654] T 39f1842ff1494d78bfa0811a799c4bb8 P e77d1c0d41034d1eb5a036dc17c071f5: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:31.101397 10254 tablet_bootstrap.cc:492] T 66ccad5a808b42378ccecc9da81cee74 P 4b7f7ba2ac084e829c4304c16a94894f: No bootstrap required, opened a new log
I20240502 01:59:31.101467 10255 tablet_bootstrap.cc:492] T 1851ea6467584b0ea32a1f68696803d0 P bdbecaeb8f214529b2012a9c003de2ff: No bootstrap required, opened a new log
I20240502 01:59:31.101477 10254 ts_tablet_manager.cc:1367] T 66ccad5a808b42378ccecc9da81cee74 P 4b7f7ba2ac084e829c4304c16a94894f: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240502 01:59:31.101563 10255 ts_tablet_manager.cc:1367] T 1851ea6467584b0ea32a1f68696803d0 P bdbecaeb8f214529b2012a9c003de2ff: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:59:31.101943 10255 raft_consensus.cc:348] T 1851ea6467584b0ea32a1f68696803d0 P bdbecaeb8f214529b2012a9c003de2ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.102067 10255 raft_consensus.cc:374] T 1851ea6467584b0ea32a1f68696803d0 P bdbecaeb8f214529b2012a9c003de2ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:31.102099 10255 raft_consensus.cc:729] T 1851ea6467584b0ea32a1f68696803d0 P bdbecaeb8f214529b2012a9c003de2ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdbecaeb8f214529b2012a9c003de2ff, State: Initialized, Role: FOLLOWER
I20240502 01:59:31.101943 10254 raft_consensus.cc:348] T 66ccad5a808b42378ccecc9da81cee74 P 4b7f7ba2ac084e829c4304c16a94894f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.102207 10255 consensus_queue.cc:260] T 1851ea6467584b0ea32a1f68696803d0 P bdbecaeb8f214529b2012a9c003de2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.102257 10254 raft_consensus.cc:374] T 66ccad5a808b42378ccecc9da81cee74 P 4b7f7ba2ac084e829c4304c16a94894f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:31.102293 10254 raft_consensus.cc:729] T 66ccad5a808b42378ccecc9da81cee74 P 4b7f7ba2ac084e829c4304c16a94894f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b7f7ba2ac084e829c4304c16a94894f, State: Initialized, Role: FOLLOWER
I20240502 01:59:31.102301 10257 tablet_bootstrap.cc:492] T 39f1842ff1494d78bfa0811a799c4bb8 P e77d1c0d41034d1eb5a036dc17c071f5: No bootstrap required, opened a new log
I20240502 01:59:31.102368 10257 ts_tablet_manager.cc:1367] T 39f1842ff1494d78bfa0811a799c4bb8 P e77d1c0d41034d1eb5a036dc17c071f5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:59:31.102438 10254 consensus_queue.cc:260] T 66ccad5a808b42378ccecc9da81cee74 P 4b7f7ba2ac084e829c4304c16a94894f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.102727 10257 raft_consensus.cc:348] T 39f1842ff1494d78bfa0811a799c4bb8 P e77d1c0d41034d1eb5a036dc17c071f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.102818 10257 raft_consensus.cc:374] T 39f1842ff1494d78bfa0811a799c4bb8 P e77d1c0d41034d1eb5a036dc17c071f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:31.102845 10254 ts_tablet_manager.cc:1398] T 66ccad5a808b42378ccecc9da81cee74 P 4b7f7ba2ac084e829c4304c16a94894f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:59:31.102846 10257 raft_consensus.cc:729] T 39f1842ff1494d78bfa0811a799c4bb8 P e77d1c0d41034d1eb5a036dc17c071f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e77d1c0d41034d1eb5a036dc17c071f5, State: Initialized, Role: FOLLOWER
I20240502 01:59:31.103053 10254 tablet_bootstrap.cc:492] T 580db11a61cb46f48d5f7507cfbdbb52 P 4b7f7ba2ac084e829c4304c16a94894f: Bootstrap starting.
I20240502 01:59:31.102447 10255 ts_tablet_manager.cc:1398] T 1851ea6467584b0ea32a1f68696803d0 P bdbecaeb8f214529b2012a9c003de2ff: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:31.103317 10255 tablet_bootstrap.cc:492] T b4409766ce994f698c31aede604a6927 P bdbecaeb8f214529b2012a9c003de2ff: Bootstrap starting.
I20240502 01:59:31.104068 10254 tablet_bootstrap.cc:654] T 580db11a61cb46f48d5f7507cfbdbb52 P 4b7f7ba2ac084e829c4304c16a94894f: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:31.104168 10255 tablet_bootstrap.cc:654] T b4409766ce994f698c31aede604a6927 P bdbecaeb8f214529b2012a9c003de2ff: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:31.103017 10257 consensus_queue.cc:260] T 39f1842ff1494d78bfa0811a799c4bb8 P e77d1c0d41034d1eb5a036dc17c071f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.104950 10257 ts_tablet_manager.cc:1398] T 39f1842ff1494d78bfa0811a799c4bb8 P e77d1c0d41034d1eb5a036dc17c071f5: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:59:31.105139 10257 tablet_bootstrap.cc:492] T abf7d8b411bb46b68ac9ae77d74f8e2b P e77d1c0d41034d1eb5a036dc17c071f5: Bootstrap starting.
I20240502 01:59:31.105360 10255 tablet_bootstrap.cc:492] T b4409766ce994f698c31aede604a6927 P bdbecaeb8f214529b2012a9c003de2ff: No bootstrap required, opened a new log
I20240502 01:59:31.105435 10255 ts_tablet_manager.cc:1367] T b4409766ce994f698c31aede604a6927 P bdbecaeb8f214529b2012a9c003de2ff: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:59:31.105855 10255 raft_consensus.cc:348] T b4409766ce994f698c31aede604a6927 P bdbecaeb8f214529b2012a9c003de2ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.105950 10255 raft_consensus.cc:374] T b4409766ce994f698c31aede604a6927 P bdbecaeb8f214529b2012a9c003de2ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:31.105976 10255 raft_consensus.cc:729] T b4409766ce994f698c31aede604a6927 P bdbecaeb8f214529b2012a9c003de2ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdbecaeb8f214529b2012a9c003de2ff, State: Initialized, Role: FOLLOWER
I20240502 01:59:31.106074 10255 consensus_queue.cc:260] T b4409766ce994f698c31aede604a6927 P bdbecaeb8f214529b2012a9c003de2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.106115 10257 tablet_bootstrap.cc:654] T abf7d8b411bb46b68ac9ae77d74f8e2b P e77d1c0d41034d1eb5a036dc17c071f5: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:31.106326 10255 ts_tablet_manager.cc:1398] T b4409766ce994f698c31aede604a6927 P bdbecaeb8f214529b2012a9c003de2ff: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:59:31.106516 10255 tablet_bootstrap.cc:492] T 580db11a61cb46f48d5f7507cfbdbb52 P bdbecaeb8f214529b2012a9c003de2ff: Bootstrap starting.
I20240502 01:59:31.107342 10255 tablet_bootstrap.cc:654] T 580db11a61cb46f48d5f7507cfbdbb52 P bdbecaeb8f214529b2012a9c003de2ff: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:31.107815 10254 tablet_bootstrap.cc:492] T 580db11a61cb46f48d5f7507cfbdbb52 P 4b7f7ba2ac084e829c4304c16a94894f: No bootstrap required, opened a new log
I20240502 01:59:31.107899 10254 ts_tablet_manager.cc:1367] T 580db11a61cb46f48d5f7507cfbdbb52 P 4b7f7ba2ac084e829c4304c16a94894f: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240502 01:59:31.108412 10257 tablet_bootstrap.cc:492] T abf7d8b411bb46b68ac9ae77d74f8e2b P e77d1c0d41034d1eb5a036dc17c071f5: No bootstrap required, opened a new log
I20240502 01:59:31.108393 10254 raft_consensus.cc:348] T 580db11a61cb46f48d5f7507cfbdbb52 P 4b7f7ba2ac084e829c4304c16a94894f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.108503 10257 ts_tablet_manager.cc:1367] T abf7d8b411bb46b68ac9ae77d74f8e2b P e77d1c0d41034d1eb5a036dc17c071f5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:59:31.108507 10254 raft_consensus.cc:374] T 580db11a61cb46f48d5f7507cfbdbb52 P 4b7f7ba2ac084e829c4304c16a94894f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:31.108664 10254 raft_consensus.cc:729] T 580db11a61cb46f48d5f7507cfbdbb52 P 4b7f7ba2ac084e829c4304c16a94894f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b7f7ba2ac084e829c4304c16a94894f, State: Initialized, Role: FOLLOWER
I20240502 01:59:31.108795 10254 consensus_queue.cc:260] T 580db11a61cb46f48d5f7507cfbdbb52 P 4b7f7ba2ac084e829c4304c16a94894f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.108978 10255 tablet_bootstrap.cc:492] T 580db11a61cb46f48d5f7507cfbdbb52 P bdbecaeb8f214529b2012a9c003de2ff: No bootstrap required, opened a new log
I20240502 01:59:31.109071 10255 ts_tablet_manager.cc:1367] T 580db11a61cb46f48d5f7507cfbdbb52 P bdbecaeb8f214529b2012a9c003de2ff: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240502 01:59:31.109076 10254 ts_tablet_manager.cc:1398] T 580db11a61cb46f48d5f7507cfbdbb52 P 4b7f7ba2ac084e829c4304c16a94894f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:31.108942 10257 raft_consensus.cc:348] T abf7d8b411bb46b68ac9ae77d74f8e2b P e77d1c0d41034d1eb5a036dc17c071f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.109375 10257 raft_consensus.cc:374] T abf7d8b411bb46b68ac9ae77d74f8e2b P e77d1c0d41034d1eb5a036dc17c071f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:31.109470 10257 raft_consensus.cc:729] T abf7d8b411bb46b68ac9ae77d74f8e2b P e77d1c0d41034d1eb5a036dc17c071f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e77d1c0d41034d1eb5a036dc17c071f5, State: Initialized, Role: FOLLOWER
I20240502 01:59:31.109473 10255 raft_consensus.cc:348] T 580db11a61cb46f48d5f7507cfbdbb52 P bdbecaeb8f214529b2012a9c003de2ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.109611 10255 raft_consensus.cc:374] T 580db11a61cb46f48d5f7507cfbdbb52 P bdbecaeb8f214529b2012a9c003de2ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:31.109638 10255 raft_consensus.cc:729] T 580db11a61cb46f48d5f7507cfbdbb52 P bdbecaeb8f214529b2012a9c003de2ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdbecaeb8f214529b2012a9c003de2ff, State: Initialized, Role: FOLLOWER
I20240502 01:59:31.109431 10254 tablet_bootstrap.cc:492] T 1851ea6467584b0ea32a1f68696803d0 P 4b7f7ba2ac084e829c4304c16a94894f: Bootstrap starting.
I20240502 01:59:31.109651 10257 consensus_queue.cc:260] T abf7d8b411bb46b68ac9ae77d74f8e2b P e77d1c0d41034d1eb5a036dc17c071f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.109735 10255 consensus_queue.cc:260] T 580db11a61cb46f48d5f7507cfbdbb52 P bdbecaeb8f214529b2012a9c003de2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.109961 10257 ts_tablet_manager.cc:1398] T abf7d8b411bb46b68ac9ae77d74f8e2b P e77d1c0d41034d1eb5a036dc17c071f5: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:59:31.109997 10255 ts_tablet_manager.cc:1398] T 580db11a61cb46f48d5f7507cfbdbb52 P bdbecaeb8f214529b2012a9c003de2ff: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:31.110127 10257 tablet_bootstrap.cc:492] T 13d726dd71b14e84bc061f537f81e91d P e77d1c0d41034d1eb5a036dc17c071f5: Bootstrap starting.
I20240502 01:59:31.110148 10255 tablet_bootstrap.cc:492] T 4885222f238c4d8ea20ebc4c7e171694 P bdbecaeb8f214529b2012a9c003de2ff: Bootstrap starting.
I20240502 01:59:31.110774 10254 tablet_bootstrap.cc:654] T 1851ea6467584b0ea32a1f68696803d0 P 4b7f7ba2ac084e829c4304c16a94894f: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:31.111019 10255 tablet_bootstrap.cc:654] T 4885222f238c4d8ea20ebc4c7e171694 P bdbecaeb8f214529b2012a9c003de2ff: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:31.111049 10257 tablet_bootstrap.cc:654] T 13d726dd71b14e84bc061f537f81e91d P e77d1c0d41034d1eb5a036dc17c071f5: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:31.112042 10255 tablet_bootstrap.cc:492] T 4885222f238c4d8ea20ebc4c7e171694 P bdbecaeb8f214529b2012a9c003de2ff: No bootstrap required, opened a new log
I20240502 01:59:31.112125 10255 ts_tablet_manager.cc:1367] T 4885222f238c4d8ea20ebc4c7e171694 P bdbecaeb8f214529b2012a9c003de2ff: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:59:31.112339 10254 tablet_bootstrap.cc:492] T 1851ea6467584b0ea32a1f68696803d0 P 4b7f7ba2ac084e829c4304c16a94894f: No bootstrap required, opened a new log
I20240502 01:59:31.112427 10254 ts_tablet_manager.cc:1367] T 1851ea6467584b0ea32a1f68696803d0 P 4b7f7ba2ac084e829c4304c16a94894f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:59:31.112605 10255 raft_consensus.cc:348] T 4885222f238c4d8ea20ebc4c7e171694 P bdbecaeb8f214529b2012a9c003de2ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.112707 10255 raft_consensus.cc:374] T 4885222f238c4d8ea20ebc4c7e171694 P bdbecaeb8f214529b2012a9c003de2ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:31.112738 10255 raft_consensus.cc:729] T 4885222f238c4d8ea20ebc4c7e171694 P bdbecaeb8f214529b2012a9c003de2ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdbecaeb8f214529b2012a9c003de2ff, State: Initialized, Role: FOLLOWER
I20240502 01:59:31.112843 10255 consensus_queue.cc:260] T 4885222f238c4d8ea20ebc4c7e171694 P bdbecaeb8f214529b2012a9c003de2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.112897 10254 raft_consensus.cc:348] T 1851ea6467584b0ea32a1f68696803d0 P 4b7f7ba2ac084e829c4304c16a94894f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.113000 10254 raft_consensus.cc:374] T 1851ea6467584b0ea32a1f68696803d0 P 4b7f7ba2ac084e829c4304c16a94894f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:31.113034 10254 raft_consensus.cc:729] T 1851ea6467584b0ea32a1f68696803d0 P 4b7f7ba2ac084e829c4304c16a94894f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b7f7ba2ac084e829c4304c16a94894f, State: Initialized, Role: FOLLOWER
I20240502 01:59:31.113109 10255 ts_tablet_manager.cc:1398] T 4885222f238c4d8ea20ebc4c7e171694 P bdbecaeb8f214529b2012a9c003de2ff: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:31.113152 10254 consensus_queue.cc:260] T 1851ea6467584b0ea32a1f68696803d0 P 4b7f7ba2ac084e829c4304c16a94894f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.113413 10254 ts_tablet_manager.cc:1398] T 1851ea6467584b0ea32a1f68696803d0 P 4b7f7ba2ac084e829c4304c16a94894f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:59:31.113571 10255 tablet_bootstrap.cc:492] T bceb844e50b84e278c0c49f4f4912e00 P bdbecaeb8f214529b2012a9c003de2ff: Bootstrap starting.
I20240502 01:59:31.113595 10254 tablet_bootstrap.cc:492] T 73ea0bfb40ad43a68a6289ff1b35d005 P 4b7f7ba2ac084e829c4304c16a94894f: Bootstrap starting.
I20240502 01:59:31.114282 10257 tablet_bootstrap.cc:492] T 13d726dd71b14e84bc061f537f81e91d P e77d1c0d41034d1eb5a036dc17c071f5: No bootstrap required, opened a new log
I20240502 01:59:31.114356 10257 ts_tablet_manager.cc:1367] T 13d726dd71b14e84bc061f537f81e91d P e77d1c0d41034d1eb5a036dc17c071f5: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240502 01:59:31.114532 10254 tablet_bootstrap.cc:654] T 73ea0bfb40ad43a68a6289ff1b35d005 P 4b7f7ba2ac084e829c4304c16a94894f: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:31.114532 10255 tablet_bootstrap.cc:654] T bceb844e50b84e278c0c49f4f4912e00 P bdbecaeb8f214529b2012a9c003de2ff: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:31.114770 10257 raft_consensus.cc:348] T 13d726dd71b14e84bc061f537f81e91d P e77d1c0d41034d1eb5a036dc17c071f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.114863 10257 raft_consensus.cc:374] T 13d726dd71b14e84bc061f537f81e91d P e77d1c0d41034d1eb5a036dc17c071f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:31.114894 10257 raft_consensus.cc:729] T 13d726dd71b14e84bc061f537f81e91d P e77d1c0d41034d1eb5a036dc17c071f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e77d1c0d41034d1eb5a036dc17c071f5, State: Initialized, Role: FOLLOWER
I20240502 01:59:31.115010 10257 consensus_queue.cc:260] T 13d726dd71b14e84bc061f537f81e91d P e77d1c0d41034d1eb5a036dc17c071f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.115255 10257 ts_tablet_manager.cc:1398] T 13d726dd71b14e84bc061f537f81e91d P e77d1c0d41034d1eb5a036dc17c071f5: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:59:31.115437 10257 tablet_bootstrap.cc:492] T 1851ea6467584b0ea32a1f68696803d0 P e77d1c0d41034d1eb5a036dc17c071f5: Bootstrap starting.
I20240502 01:59:31.116292 10255 tablet_bootstrap.cc:492] T bceb844e50b84e278c0c49f4f4912e00 P bdbecaeb8f214529b2012a9c003de2ff: No bootstrap required, opened a new log
I20240502 01:59:31.116362 10257 tablet_bootstrap.cc:654] T 1851ea6467584b0ea32a1f68696803d0 P e77d1c0d41034d1eb5a036dc17c071f5: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:31.116384 10255 ts_tablet_manager.cc:1367] T bceb844e50b84e278c0c49f4f4912e00 P bdbecaeb8f214529b2012a9c003de2ff: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:59:31.116452 10254 tablet_bootstrap.cc:492] T 73ea0bfb40ad43a68a6289ff1b35d005 P 4b7f7ba2ac084e829c4304c16a94894f: No bootstrap required, opened a new log
I20240502 01:59:31.116520 10254 ts_tablet_manager.cc:1367] T 73ea0bfb40ad43a68a6289ff1b35d005 P 4b7f7ba2ac084e829c4304c16a94894f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:59:31.116813 10255 raft_consensus.cc:348] T bceb844e50b84e278c0c49f4f4912e00 P bdbecaeb8f214529b2012a9c003de2ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.116904 10254 raft_consensus.cc:348] T 73ea0bfb40ad43a68a6289ff1b35d005 P 4b7f7ba2ac084e829c4304c16a94894f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.116998 10254 raft_consensus.cc:374] T 73ea0bfb40ad43a68a6289ff1b35d005 P 4b7f7ba2ac084e829c4304c16a94894f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:31.117029 10254 raft_consensus.cc:729] T 73ea0bfb40ad43a68a6289ff1b35d005 P 4b7f7ba2ac084e829c4304c16a94894f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b7f7ba2ac084e829c4304c16a94894f, State: Initialized, Role: FOLLOWER
I20240502 01:59:31.116914 10255 raft_consensus.cc:374] T bceb844e50b84e278c0c49f4f4912e00 P bdbecaeb8f214529b2012a9c003de2ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:31.117161 10254 consensus_queue.cc:260] T 73ea0bfb40ad43a68a6289ff1b35d005 P 4b7f7ba2ac084e829c4304c16a94894f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.117214 10255 raft_consensus.cc:729] T bceb844e50b84e278c0c49f4f4912e00 P bdbecaeb8f214529b2012a9c003de2ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdbecaeb8f214529b2012a9c003de2ff, State: Initialized, Role: FOLLOWER
I20240502 01:59:31.117384 10257 tablet_bootstrap.cc:492] T 1851ea6467584b0ea32a1f68696803d0 P e77d1c0d41034d1eb5a036dc17c071f5: No bootstrap required, opened a new log
I20240502 01:59:31.117446 10254 ts_tablet_manager.cc:1398] T 73ea0bfb40ad43a68a6289ff1b35d005 P 4b7f7ba2ac084e829c4304c16a94894f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:59:31.117519 10257 ts_tablet_manager.cc:1367] T 1851ea6467584b0ea32a1f68696803d0 P e77d1c0d41034d1eb5a036dc17c071f5: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:59:31.117560 10255 consensus_queue.cc:260] T bceb844e50b84e278c0c49f4f4912e00 P bdbecaeb8f214529b2012a9c003de2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.117621 10254 tablet_bootstrap.cc:492] T 39f1842ff1494d78bfa0811a799c4bb8 P 4b7f7ba2ac084e829c4304c16a94894f: Bootstrap starting.
I20240502 01:59:31.117903 10257 raft_consensus.cc:348] T 1851ea6467584b0ea32a1f68696803d0 P e77d1c0d41034d1eb5a036dc17c071f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.117992 10257 raft_consensus.cc:374] T 1851ea6467584b0ea32a1f68696803d0 P e77d1c0d41034d1eb5a036dc17c071f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:31.118021 10257 raft_consensus.cc:729] T 1851ea6467584b0ea32a1f68696803d0 P e77d1c0d41034d1eb5a036dc17c071f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e77d1c0d41034d1eb5a036dc17c071f5, State: Initialized, Role: FOLLOWER
I20240502 01:59:31.118119 10257 consensus_queue.cc:260] T 1851ea6467584b0ea32a1f68696803d0 P e77d1c0d41034d1eb5a036dc17c071f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.118358 10257 ts_tablet_manager.cc:1398] T 1851ea6467584b0ea32a1f68696803d0 P e77d1c0d41034d1eb5a036dc17c071f5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:59:31.118515 10257 tablet_bootstrap.cc:492] T 580db11a61cb46f48d5f7507cfbdbb52 P e77d1c0d41034d1eb5a036dc17c071f5: Bootstrap starting.
I20240502 01:59:31.118618 10254 tablet_bootstrap.cc:654] T 39f1842ff1494d78bfa0811a799c4bb8 P 4b7f7ba2ac084e829c4304c16a94894f: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:31.119370 10257 tablet_bootstrap.cc:654] T 580db11a61cb46f48d5f7507cfbdbb52 P e77d1c0d41034d1eb5a036dc17c071f5: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:31.119463 10255 ts_tablet_manager.cc:1398] T bceb844e50b84e278c0c49f4f4912e00 P bdbecaeb8f214529b2012a9c003de2ff: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:59:31.119644 10255 tablet_bootstrap.cc:492] T abf7d8b411bb46b68ac9ae77d74f8e2b P bdbecaeb8f214529b2012a9c003de2ff: Bootstrap starting.
I20240502 01:59:31.120414 10257 tablet_bootstrap.cc:492] T 580db11a61cb46f48d5f7507cfbdbb52 P e77d1c0d41034d1eb5a036dc17c071f5: No bootstrap required, opened a new log
I20240502 01:59:31.120508 10257 ts_tablet_manager.cc:1367] T 580db11a61cb46f48d5f7507cfbdbb52 P e77d1c0d41034d1eb5a036dc17c071f5: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:59:31.120563 10254 tablet_bootstrap.cc:492] T 39f1842ff1494d78bfa0811a799c4bb8 P 4b7f7ba2ac084e829c4304c16a94894f: No bootstrap required, opened a new log
I20240502 01:59:31.120623 10254 ts_tablet_manager.cc:1367] T 39f1842ff1494d78bfa0811a799c4bb8 P 4b7f7ba2ac084e829c4304c16a94894f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:59:31.120649 10255 tablet_bootstrap.cc:654] T abf7d8b411bb46b68ac9ae77d74f8e2b P bdbecaeb8f214529b2012a9c003de2ff: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:31.120896 10257 raft_consensus.cc:348] T 580db11a61cb46f48d5f7507cfbdbb52 P e77d1c0d41034d1eb5a036dc17c071f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.120934 10254 raft_consensus.cc:348] T 39f1842ff1494d78bfa0811a799c4bb8 P 4b7f7ba2ac084e829c4304c16a94894f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.120988 10257 raft_consensus.cc:374] T 580db11a61cb46f48d5f7507cfbdbb52 P e77d1c0d41034d1eb5a036dc17c071f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:31.121050 10257 raft_consensus.cc:729] T 580db11a61cb46f48d5f7507cfbdbb52 P e77d1c0d41034d1eb5a036dc17c071f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e77d1c0d41034d1eb5a036dc17c071f5, State: Initialized, Role: FOLLOWER
I20240502 01:59:31.121156 10257 consensus_queue.cc:260] T 580db11a61cb46f48d5f7507cfbdbb52 P e77d1c0d41034d1eb5a036dc17c071f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.121395 10257 ts_tablet_manager.cc:1398] T 580db11a61cb46f48d5f7507cfbdbb52 P e77d1c0d41034d1eb5a036dc17c071f5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:59:31.121008 10254 raft_consensus.cc:374] T 39f1842ff1494d78bfa0811a799c4bb8 P 4b7f7ba2ac084e829c4304c16a94894f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:31.121549 10254 raft_consensus.cc:729] T 39f1842ff1494d78bfa0811a799c4bb8 P 4b7f7ba2ac084e829c4304c16a94894f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b7f7ba2ac084e829c4304c16a94894f, State: Initialized, Role: FOLLOWER
I20240502 01:59:31.121601 10257 tablet_bootstrap.cc:492] T 19ff9c216a094fde86ea9f9648bf08cb P e77d1c0d41034d1eb5a036dc17c071f5: Bootstrap starting.
I20240502 01:59:31.121656 10254 consensus_queue.cc:260] T 39f1842ff1494d78bfa0811a799c4bb8 P 4b7f7ba2ac084e829c4304c16a94894f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.121709 10255 tablet_bootstrap.cc:492] T abf7d8b411bb46b68ac9ae77d74f8e2b P bdbecaeb8f214529b2012a9c003de2ff: No bootstrap required, opened a new log
I20240502 01:59:31.121780 10255 ts_tablet_manager.cc:1367] T abf7d8b411bb46b68ac9ae77d74f8e2b P bdbecaeb8f214529b2012a9c003de2ff: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:59:31.121891 10254 ts_tablet_manager.cc:1398] T 39f1842ff1494d78bfa0811a799c4bb8 P 4b7f7ba2ac084e829c4304c16a94894f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:31.122181 10255 raft_consensus.cc:348] T abf7d8b411bb46b68ac9ae77d74f8e2b P bdbecaeb8f214529b2012a9c003de2ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.122344 10255 raft_consensus.cc:374] T abf7d8b411bb46b68ac9ae77d74f8e2b P bdbecaeb8f214529b2012a9c003de2ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:31.122433 10255 raft_consensus.cc:729] T abf7d8b411bb46b68ac9ae77d74f8e2b P bdbecaeb8f214529b2012a9c003de2ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdbecaeb8f214529b2012a9c003de2ff, State: Initialized, Role: FOLLOWER
I20240502 01:59:31.122526 10257 tablet_bootstrap.cc:654] T 19ff9c216a094fde86ea9f9648bf08cb P e77d1c0d41034d1eb5a036dc17c071f5: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:31.122251 10254 tablet_bootstrap.cc:492] T 13d726dd71b14e84bc061f537f81e91d P 4b7f7ba2ac084e829c4304c16a94894f: Bootstrap starting.
I20240502 01:59:31.123505 10254 tablet_bootstrap.cc:654] T 13d726dd71b14e84bc061f537f81e91d P 4b7f7ba2ac084e829c4304c16a94894f: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:31.123570 10257 tablet_bootstrap.cc:492] T 19ff9c216a094fde86ea9f9648bf08cb P e77d1c0d41034d1eb5a036dc17c071f5: No bootstrap required, opened a new log
I20240502 01:59:31.123651 10257 ts_tablet_manager.cc:1367] T 19ff9c216a094fde86ea9f9648bf08cb P e77d1c0d41034d1eb5a036dc17c071f5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:59:31.124061 10257 raft_consensus.cc:348] T 19ff9c216a094fde86ea9f9648bf08cb P e77d1c0d41034d1eb5a036dc17c071f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.124150 10257 raft_consensus.cc:374] T 19ff9c216a094fde86ea9f9648bf08cb P e77d1c0d41034d1eb5a036dc17c071f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:31.124132 10255 consensus_queue.cc:260] T abf7d8b411bb46b68ac9ae77d74f8e2b P bdbecaeb8f214529b2012a9c003de2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.124223 10257 raft_consensus.cc:729] T 19ff9c216a094fde86ea9f9648bf08cb P e77d1c0d41034d1eb5a036dc17c071f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e77d1c0d41034d1eb5a036dc17c071f5, State: Initialized, Role: FOLLOWER
I20240502 01:59:31.124389 10257 consensus_queue.cc:260] T 19ff9c216a094fde86ea9f9648bf08cb P e77d1c0d41034d1eb5a036dc17c071f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.124560 10255 ts_tablet_manager.cc:1398] T abf7d8b411bb46b68ac9ae77d74f8e2b P bdbecaeb8f214529b2012a9c003de2ff: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240502 01:59:31.124760 10257 ts_tablet_manager.cc:1398] T 19ff9c216a094fde86ea9f9648bf08cb P e77d1c0d41034d1eb5a036dc17c071f5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:59:31.125012 10257 tablet_bootstrap.cc:492] T 73ea0bfb40ad43a68a6289ff1b35d005 P e77d1c0d41034d1eb5a036dc17c071f5: Bootstrap starting.
I20240502 01:59:31.125530 10254 tablet_bootstrap.cc:492] T 13d726dd71b14e84bc061f537f81e91d P 4b7f7ba2ac084e829c4304c16a94894f: No bootstrap required, opened a new log
I20240502 01:59:31.125681 10254 ts_tablet_manager.cc:1367] T 13d726dd71b14e84bc061f537f81e91d P 4b7f7ba2ac084e829c4304c16a94894f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:59:31.125916 10257 tablet_bootstrap.cc:654] T 73ea0bfb40ad43a68a6289ff1b35d005 P e77d1c0d41034d1eb5a036dc17c071f5: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:31.126276 10254 raft_consensus.cc:348] T 13d726dd71b14e84bc061f537f81e91d P 4b7f7ba2ac084e829c4304c16a94894f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.124768 10255 tablet_bootstrap.cc:492] T e7a8fc5afcab4b1db4ebf245b66df4be P bdbecaeb8f214529b2012a9c003de2ff: Bootstrap starting.
I20240502 01:59:31.126444 10254 raft_consensus.cc:374] T 13d726dd71b14e84bc061f537f81e91d P 4b7f7ba2ac084e829c4304c16a94894f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:31.126554 10254 raft_consensus.cc:729] T 13d726dd71b14e84bc061f537f81e91d P 4b7f7ba2ac084e829c4304c16a94894f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b7f7ba2ac084e829c4304c16a94894f, State: Initialized, Role: FOLLOWER
I20240502 01:59:31.126715 10254 consensus_queue.cc:260] T 13d726dd71b14e84bc061f537f81e91d P 4b7f7ba2ac084e829c4304c16a94894f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.127120 10254 ts_tablet_manager.cc:1398] T 13d726dd71b14e84bc061f537f81e91d P 4b7f7ba2ac084e829c4304c16a94894f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:31.127280 10255 tablet_bootstrap.cc:654] T e7a8fc5afcab4b1db4ebf245b66df4be P bdbecaeb8f214529b2012a9c003de2ff: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:31.127300 10254 tablet_bootstrap.cc:492] T e7a8fc5afcab4b1db4ebf245b66df4be P 4b7f7ba2ac084e829c4304c16a94894f: Bootstrap starting.
I20240502 01:59:31.127388 10257 tablet_bootstrap.cc:492] T 73ea0bfb40ad43a68a6289ff1b35d005 P e77d1c0d41034d1eb5a036dc17c071f5: No bootstrap required, opened a new log
I20240502 01:59:31.127457 10257 ts_tablet_manager.cc:1367] T 73ea0bfb40ad43a68a6289ff1b35d005 P e77d1c0d41034d1eb5a036dc17c071f5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:59:31.127897 10257 raft_consensus.cc:348] T 73ea0bfb40ad43a68a6289ff1b35d005 P e77d1c0d41034d1eb5a036dc17c071f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.128031 10257 raft_consensus.cc:374] T 73ea0bfb40ad43a68a6289ff1b35d005 P e77d1c0d41034d1eb5a036dc17c071f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:31.128087 10257 raft_consensus.cc:729] T 73ea0bfb40ad43a68a6289ff1b35d005 P e77d1c0d41034d1eb5a036dc17c071f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e77d1c0d41034d1eb5a036dc17c071f5, State: Initialized, Role: FOLLOWER
I20240502 01:59:31.128233 10254 tablet_bootstrap.cc:654] T e7a8fc5afcab4b1db4ebf245b66df4be P 4b7f7ba2ac084e829c4304c16a94894f: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:31.128310 10257 consensus_queue.cc:260] T 73ea0bfb40ad43a68a6289ff1b35d005 P e77d1c0d41034d1eb5a036dc17c071f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.128602 10257 ts_tablet_manager.cc:1398] T 73ea0bfb40ad43a68a6289ff1b35d005 P e77d1c0d41034d1eb5a036dc17c071f5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:31.128763 10257 tablet_bootstrap.cc:492] T bceb844e50b84e278c0c49f4f4912e00 P e77d1c0d41034d1eb5a036dc17c071f5: Bootstrap starting.
I20240502 01:59:31.129381 10255 tablet_bootstrap.cc:492] T e7a8fc5afcab4b1db4ebf245b66df4be P bdbecaeb8f214529b2012a9c003de2ff: No bootstrap required, opened a new log
I20240502 01:59:31.129459 10255 ts_tablet_manager.cc:1367] T e7a8fc5afcab4b1db4ebf245b66df4be P bdbecaeb8f214529b2012a9c003de2ff: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20240502 01:59:31.129673 10257 tablet_bootstrap.cc:654] T bceb844e50b84e278c0c49f4f4912e00 P e77d1c0d41034d1eb5a036dc17c071f5: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:31.129906 10255 raft_consensus.cc:348] T e7a8fc5afcab4b1db4ebf245b66df4be P bdbecaeb8f214529b2012a9c003de2ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.130051 10255 raft_consensus.cc:374] T e7a8fc5afcab4b1db4ebf245b66df4be P bdbecaeb8f214529b2012a9c003de2ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:31.130128 10255 raft_consensus.cc:729] T e7a8fc5afcab4b1db4ebf245b66df4be P bdbecaeb8f214529b2012a9c003de2ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdbecaeb8f214529b2012a9c003de2ff, State: Initialized, Role: FOLLOWER
I20240502 01:59:31.130292 10255 consensus_queue.cc:260] T e7a8fc5afcab4b1db4ebf245b66df4be P bdbecaeb8f214529b2012a9c003de2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.130614 10255 ts_tablet_manager.cc:1398] T e7a8fc5afcab4b1db4ebf245b66df4be P bdbecaeb8f214529b2012a9c003de2ff: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:31.130697 10254 tablet_bootstrap.cc:492] T e7a8fc5afcab4b1db4ebf245b66df4be P 4b7f7ba2ac084e829c4304c16a94894f: No bootstrap required, opened a new log
I20240502 01:59:31.130779 10254 ts_tablet_manager.cc:1367] T e7a8fc5afcab4b1db4ebf245b66df4be P 4b7f7ba2ac084e829c4304c16a94894f: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240502 01:59:31.130832 10255 tablet_bootstrap.cc:492] T 39f1842ff1494d78bfa0811a799c4bb8 P bdbecaeb8f214529b2012a9c003de2ff: Bootstrap starting.
I20240502 01:59:31.130914 10257 tablet_bootstrap.cc:492] T bceb844e50b84e278c0c49f4f4912e00 P e77d1c0d41034d1eb5a036dc17c071f5: No bootstrap required, opened a new log
I20240502 01:59:31.130999 10257 ts_tablet_manager.cc:1367] T bceb844e50b84e278c0c49f4f4912e00 P e77d1c0d41034d1eb5a036dc17c071f5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:59:31.131170 10254 raft_consensus.cc:348] T e7a8fc5afcab4b1db4ebf245b66df4be P 4b7f7ba2ac084e829c4304c16a94894f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.131280 10254 raft_consensus.cc:374] T e7a8fc5afcab4b1db4ebf245b66df4be P 4b7f7ba2ac084e829c4304c16a94894f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:31.131311 10254 raft_consensus.cc:729] T e7a8fc5afcab4b1db4ebf245b66df4be P 4b7f7ba2ac084e829c4304c16a94894f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b7f7ba2ac084e829c4304c16a94894f, State: Initialized, Role: FOLLOWER
I20240502 01:59:31.131374 10257 raft_consensus.cc:348] T bceb844e50b84e278c0c49f4f4912e00 P e77d1c0d41034d1eb5a036dc17c071f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.131424 10254 consensus_queue.cc:260] T e7a8fc5afcab4b1db4ebf245b66df4be P 4b7f7ba2ac084e829c4304c16a94894f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.131464 10257 raft_consensus.cc:374] T bceb844e50b84e278c0c49f4f4912e00 P e77d1c0d41034d1eb5a036dc17c071f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:31.131553 10257 raft_consensus.cc:729] T bceb844e50b84e278c0c49f4f4912e00 P e77d1c0d41034d1eb5a036dc17c071f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e77d1c0d41034d1eb5a036dc17c071f5, State: Initialized, Role: FOLLOWER
I20240502 01:59:31.131664 10257 consensus_queue.cc:260] T bceb844e50b84e278c0c49f4f4912e00 P e77d1c0d41034d1eb5a036dc17c071f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.131716 10254 ts_tablet_manager.cc:1398] T e7a8fc5afcab4b1db4ebf245b66df4be P 4b7f7ba2ac084e829c4304c16a94894f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:59:31.131950 10254 tablet_bootstrap.cc:492] T 19ff9c216a094fde86ea9f9648bf08cb P 4b7f7ba2ac084e829c4304c16a94894f: Bootstrap starting.
I20240502 01:59:31.131728 10255 tablet_bootstrap.cc:654] T 39f1842ff1494d78bfa0811a799c4bb8 P bdbecaeb8f214529b2012a9c003de2ff: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:31.132454 10257 ts_tablet_manager.cc:1398] T bceb844e50b84e278c0c49f4f4912e00 P e77d1c0d41034d1eb5a036dc17c071f5: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:59:31.132701 10257 tablet_bootstrap.cc:492] T e7a8fc5afcab4b1db4ebf245b66df4be P e77d1c0d41034d1eb5a036dc17c071f5: Bootstrap starting.
I20240502 01:59:31.132938 10254 tablet_bootstrap.cc:654] T 19ff9c216a094fde86ea9f9648bf08cb P 4b7f7ba2ac084e829c4304c16a94894f: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:31.133615 10257 tablet_bootstrap.cc:654] T e7a8fc5afcab4b1db4ebf245b66df4be P e77d1c0d41034d1eb5a036dc17c071f5: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:31.133934 10255 tablet_bootstrap.cc:492] T 39f1842ff1494d78bfa0811a799c4bb8 P bdbecaeb8f214529b2012a9c003de2ff: No bootstrap required, opened a new log
I20240502 01:59:31.134021 10255 ts_tablet_manager.cc:1367] T 39f1842ff1494d78bfa0811a799c4bb8 P bdbecaeb8f214529b2012a9c003de2ff: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:59:31.134486 10255 raft_consensus.cc:348] T 39f1842ff1494d78bfa0811a799c4bb8 P bdbecaeb8f214529b2012a9c003de2ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.134593 10255 raft_consensus.cc:374] T 39f1842ff1494d78bfa0811a799c4bb8 P bdbecaeb8f214529b2012a9c003de2ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:31.134627 10255 raft_consensus.cc:729] T 39f1842ff1494d78bfa0811a799c4bb8 P bdbecaeb8f214529b2012a9c003de2ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdbecaeb8f214529b2012a9c003de2ff, State: Initialized, Role: FOLLOWER
I20240502 01:59:31.134750 10255 consensus_queue.cc:260] T 39f1842ff1494d78bfa0811a799c4bb8 P bdbecaeb8f214529b2012a9c003de2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.135037 10255 ts_tablet_manager.cc:1398] T 39f1842ff1494d78bfa0811a799c4bb8 P bdbecaeb8f214529b2012a9c003de2ff: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:31.135221 10255 tablet_bootstrap.cc:492] T 19ff9c216a094fde86ea9f9648bf08cb P bdbecaeb8f214529b2012a9c003de2ff: Bootstrap starting.
I20240502 01:59:31.136129 10255 tablet_bootstrap.cc:654] T 19ff9c216a094fde86ea9f9648bf08cb P bdbecaeb8f214529b2012a9c003de2ff: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:31.136339 10254 tablet_bootstrap.cc:492] T 19ff9c216a094fde86ea9f9648bf08cb P 4b7f7ba2ac084e829c4304c16a94894f: No bootstrap required, opened a new log
I20240502 01:59:31.136339 10257 tablet_bootstrap.cc:492] T e7a8fc5afcab4b1db4ebf245b66df4be P e77d1c0d41034d1eb5a036dc17c071f5: No bootstrap required, opened a new log
I20240502 01:59:31.136432 10257 ts_tablet_manager.cc:1367] T e7a8fc5afcab4b1db4ebf245b66df4be P e77d1c0d41034d1eb5a036dc17c071f5: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20240502 01:59:31.136432 10254 ts_tablet_manager.cc:1367] T 19ff9c216a094fde86ea9f9648bf08cb P 4b7f7ba2ac084e829c4304c16a94894f: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:59:31.136888 10254 raft_consensus.cc:348] T 19ff9c216a094fde86ea9f9648bf08cb P 4b7f7ba2ac084e829c4304c16a94894f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.136866 10257 raft_consensus.cc:348] T e7a8fc5afcab4b1db4ebf245b66df4be P e77d1c0d41034d1eb5a036dc17c071f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.136981 10254 raft_consensus.cc:374] T 19ff9c216a094fde86ea9f9648bf08cb P 4b7f7ba2ac084e829c4304c16a94894f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:31.137012 10254 raft_consensus.cc:729] T 19ff9c216a094fde86ea9f9648bf08cb P 4b7f7ba2ac084e829c4304c16a94894f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b7f7ba2ac084e829c4304c16a94894f, State: Initialized, Role: FOLLOWER
I20240502 01:59:31.137012 10257 raft_consensus.cc:374] T e7a8fc5afcab4b1db4ebf245b66df4be P e77d1c0d41034d1eb5a036dc17c071f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:31.137068 10257 raft_consensus.cc:729] T e7a8fc5afcab4b1db4ebf245b66df4be P e77d1c0d41034d1eb5a036dc17c071f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e77d1c0d41034d1eb5a036dc17c071f5, State: Initialized, Role: FOLLOWER
I20240502 01:59:31.137141 10254 consensus_queue.cc:260] T 19ff9c216a094fde86ea9f9648bf08cb P 4b7f7ba2ac084e829c4304c16a94894f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.137185 10257 consensus_queue.cc:260] T e7a8fc5afcab4b1db4ebf245b66df4be P e77d1c0d41034d1eb5a036dc17c071f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.137432 10254 ts_tablet_manager.cc:1398] T 19ff9c216a094fde86ea9f9648bf08cb P 4b7f7ba2ac084e829c4304c16a94894f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:31.137516 10255 tablet_bootstrap.cc:492] T 19ff9c216a094fde86ea9f9648bf08cb P bdbecaeb8f214529b2012a9c003de2ff: No bootstrap required, opened a new log
I20240502 01:59:31.137584 10254 tablet_bootstrap.cc:492] T bceb844e50b84e278c0c49f4f4912e00 P 4b7f7ba2ac084e829c4304c16a94894f: Bootstrap starting.
I20240502 01:59:31.137594 10255 ts_tablet_manager.cc:1367] T 19ff9c216a094fde86ea9f9648bf08cb P bdbecaeb8f214529b2012a9c003de2ff: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:59:31.138028 10257 ts_tablet_manager.cc:1398] T e7a8fc5afcab4b1db4ebf245b66df4be P e77d1c0d41034d1eb5a036dc17c071f5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:59:31.138211 10257 tablet_bootstrap.cc:492] T 66ccad5a808b42378ccecc9da81cee74 P e77d1c0d41034d1eb5a036dc17c071f5: Bootstrap starting.
I20240502 01:59:31.138211 10255 raft_consensus.cc:348] T 19ff9c216a094fde86ea9f9648bf08cb P bdbecaeb8f214529b2012a9c003de2ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.138345 10255 raft_consensus.cc:374] T 19ff9c216a094fde86ea9f9648bf08cb P bdbecaeb8f214529b2012a9c003de2ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:31.138384 10255 raft_consensus.cc:729] T 19ff9c216a094fde86ea9f9648bf08cb P bdbecaeb8f214529b2012a9c003de2ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdbecaeb8f214529b2012a9c003de2ff, State: Initialized, Role: FOLLOWER
I20240502 01:59:31.138451 10254 tablet_bootstrap.cc:654] T bceb844e50b84e278c0c49f4f4912e00 P 4b7f7ba2ac084e829c4304c16a94894f: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:31.138484 10255 consensus_queue.cc:260] T 19ff9c216a094fde86ea9f9648bf08cb P bdbecaeb8f214529b2012a9c003de2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.138803 10255 ts_tablet_manager.cc:1398] T 19ff9c216a094fde86ea9f9648bf08cb P bdbecaeb8f214529b2012a9c003de2ff: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:59:31.138975 10255 tablet_bootstrap.cc:492] T 13d726dd71b14e84bc061f537f81e91d P bdbecaeb8f214529b2012a9c003de2ff: Bootstrap starting.
I20240502 01:59:31.139122 10257 tablet_bootstrap.cc:654] T 66ccad5a808b42378ccecc9da81cee74 P e77d1c0d41034d1eb5a036dc17c071f5: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:31.139901 10255 tablet_bootstrap.cc:654] T 13d726dd71b14e84bc061f537f81e91d P bdbecaeb8f214529b2012a9c003de2ff: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:31.140093 10257 tablet_bootstrap.cc:492] T 66ccad5a808b42378ccecc9da81cee74 P e77d1c0d41034d1eb5a036dc17c071f5: No bootstrap required, opened a new log
I20240502 01:59:31.140163 10257 ts_tablet_manager.cc:1367] T 66ccad5a808b42378ccecc9da81cee74 P e77d1c0d41034d1eb5a036dc17c071f5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:59:31.140566 10257 raft_consensus.cc:348] T 66ccad5a808b42378ccecc9da81cee74 P e77d1c0d41034d1eb5a036dc17c071f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.140650 10257 raft_consensus.cc:374] T 66ccad5a808b42378ccecc9da81cee74 P e77d1c0d41034d1eb5a036dc17c071f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:31.140676 10257 raft_consensus.cc:729] T 66ccad5a808b42378ccecc9da81cee74 P e77d1c0d41034d1eb5a036dc17c071f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e77d1c0d41034d1eb5a036dc17c071f5, State: Initialized, Role: FOLLOWER
I20240502 01:59:31.140772 10257 consensus_queue.cc:260] T 66ccad5a808b42378ccecc9da81cee74 P e77d1c0d41034d1eb5a036dc17c071f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.141088 10257 ts_tablet_manager.cc:1398] T 66ccad5a808b42378ccecc9da81cee74 P e77d1c0d41034d1eb5a036dc17c071f5: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:59:31.141299 10257 tablet_bootstrap.cc:492] T b4409766ce994f698c31aede604a6927 P e77d1c0d41034d1eb5a036dc17c071f5: Bootstrap starting.
I20240502 01:59:31.142186 10257 tablet_bootstrap.cc:654] T b4409766ce994f698c31aede604a6927 P e77d1c0d41034d1eb5a036dc17c071f5: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:31.142482 10254 tablet_bootstrap.cc:492] T bceb844e50b84e278c0c49f4f4912e00 P 4b7f7ba2ac084e829c4304c16a94894f: No bootstrap required, opened a new log
I20240502 01:59:31.142485 10255 tablet_bootstrap.cc:492] T 13d726dd71b14e84bc061f537f81e91d P bdbecaeb8f214529b2012a9c003de2ff: No bootstrap required, opened a new log
I20240502 01:59:31.142567 10254 ts_tablet_manager.cc:1367] T bceb844e50b84e278c0c49f4f4912e00 P 4b7f7ba2ac084e829c4304c16a94894f: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:59:31.142627 10255 ts_tablet_manager.cc:1367] T 13d726dd71b14e84bc061f537f81e91d P bdbecaeb8f214529b2012a9c003de2ff: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:59:31.142910 10254 raft_consensus.cc:348] T bceb844e50b84e278c0c49f4f4912e00 P 4b7f7ba2ac084e829c4304c16a94894f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.142993 10254 raft_consensus.cc:374] T bceb844e50b84e278c0c49f4f4912e00 P 4b7f7ba2ac084e829c4304c16a94894f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:31.143013 10254 raft_consensus.cc:729] T bceb844e50b84e278c0c49f4f4912e00 P 4b7f7ba2ac084e829c4304c16a94894f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b7f7ba2ac084e829c4304c16a94894f, State: Initialized, Role: FOLLOWER
I20240502 01:59:31.143091 10254 consensus_queue.cc:260] T bceb844e50b84e278c0c49f4f4912e00 P 4b7f7ba2ac084e829c4304c16a94894f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.143082 10255 raft_consensus.cc:348] T 13d726dd71b14e84bc061f537f81e91d P bdbecaeb8f214529b2012a9c003de2ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.143199 10255 raft_consensus.cc:374] T 13d726dd71b14e84bc061f537f81e91d P bdbecaeb8f214529b2012a9c003de2ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:31.143236 10255 raft_consensus.cc:729] T 13d726dd71b14e84bc061f537f81e91d P bdbecaeb8f214529b2012a9c003de2ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdbecaeb8f214529b2012a9c003de2ff, State: Initialized, Role: FOLLOWER
I20240502 01:59:31.143344 10254 ts_tablet_manager.cc:1398] T bceb844e50b84e278c0c49f4f4912e00 P 4b7f7ba2ac084e829c4304c16a94894f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:31.143829 10257 tablet_bootstrap.cc:492] T b4409766ce994f698c31aede604a6927 P e77d1c0d41034d1eb5a036dc17c071f5: No bootstrap required, opened a new log
I20240502 01:59:31.143357 10255 consensus_queue.cc:260] T 13d726dd71b14e84bc061f537f81e91d P bdbecaeb8f214529b2012a9c003de2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.143906 10257 ts_tablet_manager.cc:1367] T b4409766ce994f698c31aede604a6927 P e77d1c0d41034d1eb5a036dc17c071f5: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240502 01:59:31.144088 10255 ts_tablet_manager.cc:1398] T 13d726dd71b14e84bc061f537f81e91d P bdbecaeb8f214529b2012a9c003de2ff: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:31.144289 10255 tablet_bootstrap.cc:492] T 73ea0bfb40ad43a68a6289ff1b35d005 P bdbecaeb8f214529b2012a9c003de2ff: Bootstrap starting.
I20240502 01:59:31.144357 10257 raft_consensus.cc:348] T b4409766ce994f698c31aede604a6927 P e77d1c0d41034d1eb5a036dc17c071f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.144443 10257 raft_consensus.cc:374] T b4409766ce994f698c31aede604a6927 P e77d1c0d41034d1eb5a036dc17c071f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:31.144475 10257 raft_consensus.cc:729] T b4409766ce994f698c31aede604a6927 P e77d1c0d41034d1eb5a036dc17c071f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e77d1c0d41034d1eb5a036dc17c071f5, State: Initialized, Role: FOLLOWER
I20240502 01:59:31.144613 10257 consensus_queue.cc:260] T b4409766ce994f698c31aede604a6927 P e77d1c0d41034d1eb5a036dc17c071f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.144892 10257 ts_tablet_manager.cc:1398] T b4409766ce994f698c31aede604a6927 P e77d1c0d41034d1eb5a036dc17c071f5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:31.145210 10255 tablet_bootstrap.cc:654] T 73ea0bfb40ad43a68a6289ff1b35d005 P bdbecaeb8f214529b2012a9c003de2ff: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:31.146508 10255 tablet_bootstrap.cc:492] T 73ea0bfb40ad43a68a6289ff1b35d005 P bdbecaeb8f214529b2012a9c003de2ff: No bootstrap required, opened a new log
I20240502 01:59:31.146590 10255 ts_tablet_manager.cc:1367] T 73ea0bfb40ad43a68a6289ff1b35d005 P bdbecaeb8f214529b2012a9c003de2ff: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:59:31.147017 10255 raft_consensus.cc:348] T 73ea0bfb40ad43a68a6289ff1b35d005 P bdbecaeb8f214529b2012a9c003de2ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.147125 10255 raft_consensus.cc:374] T 73ea0bfb40ad43a68a6289ff1b35d005 P bdbecaeb8f214529b2012a9c003de2ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:31.147156 10255 raft_consensus.cc:729] T 73ea0bfb40ad43a68a6289ff1b35d005 P bdbecaeb8f214529b2012a9c003de2ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdbecaeb8f214529b2012a9c003de2ff, State: Initialized, Role: FOLLOWER
I20240502 01:59:31.147281 10255 consensus_queue.cc:260] T 73ea0bfb40ad43a68a6289ff1b35d005 P bdbecaeb8f214529b2012a9c003de2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.147553 10255 ts_tablet_manager.cc:1398] T 73ea0bfb40ad43a68a6289ff1b35d005 P bdbecaeb8f214529b2012a9c003de2ff: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:59:31.160960 10263 raft_consensus.cc:482] T 39f1842ff1494d78bfa0811a799c4bb8 P bdbecaeb8f214529b2012a9c003de2ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:31.161087 10263 raft_consensus.cc:504] T 39f1842ff1494d78bfa0811a799c4bb8 P bdbecaeb8f214529b2012a9c003de2ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.161578 10263 leader_election.cc:288] T 39f1842ff1494d78bfa0811a799c4bb8 P bdbecaeb8f214529b2012a9c003de2ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4b7f7ba2ac084e829c4304c16a94894f (127.6.117.194:46805), e77d1c0d41034d1eb5a036dc17c071f5 (127.6.117.195:43977)
I20240502 01:59:31.164989 10133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39f1842ff1494d78bfa0811a799c4bb8" candidate_uuid: "bdbecaeb8f214529b2012a9c003de2ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b7f7ba2ac084e829c4304c16a94894f" is_pre_election: true
I20240502 01:59:31.164991 10208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39f1842ff1494d78bfa0811a799c4bb8" candidate_uuid: "bdbecaeb8f214529b2012a9c003de2ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" is_pre_election: true
I20240502 01:59:31.165154 10133 raft_consensus.cc:2454] T 39f1842ff1494d78bfa0811a799c4bb8 P 4b7f7ba2ac084e829c4304c16a94894f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bdbecaeb8f214529b2012a9c003de2ff in term 0.
I20240502 01:59:31.165186 10208 raft_consensus.cc:2454] T 39f1842ff1494d78bfa0811a799c4bb8 P e77d1c0d41034d1eb5a036dc17c071f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bdbecaeb8f214529b2012a9c003de2ff in term 0.
I20240502 01:59:31.165433 10022 leader_election.cc:302] T 39f1842ff1494d78bfa0811a799c4bb8 P bdbecaeb8f214529b2012a9c003de2ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b7f7ba2ac084e829c4304c16a94894f, bdbecaeb8f214529b2012a9c003de2ff; no voters:
I20240502 01:59:31.165601 10263 raft_consensus.cc:2781] T 39f1842ff1494d78bfa0811a799c4bb8 P bdbecaeb8f214529b2012a9c003de2ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:31.165661 10263 raft_consensus.cc:482] T 39f1842ff1494d78bfa0811a799c4bb8 P bdbecaeb8f214529b2012a9c003de2ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:31.165707 10263 raft_consensus.cc:3037] T 39f1842ff1494d78bfa0811a799c4bb8 P bdbecaeb8f214529b2012a9c003de2ff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:31.166493 10263 raft_consensus.cc:504] T 39f1842ff1494d78bfa0811a799c4bb8 P bdbecaeb8f214529b2012a9c003de2ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.166836 10263 leader_election.cc:288] T 39f1842ff1494d78bfa0811a799c4bb8 P bdbecaeb8f214529b2012a9c003de2ff [CANDIDATE]: Term 1 election: Requested vote from peers 4b7f7ba2ac084e829c4304c16a94894f (127.6.117.194:46805), e77d1c0d41034d1eb5a036dc17c071f5 (127.6.117.195:43977)
I20240502 01:59:31.167129 10133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39f1842ff1494d78bfa0811a799c4bb8" candidate_uuid: "bdbecaeb8f214529b2012a9c003de2ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b7f7ba2ac084e829c4304c16a94894f"
I20240502 01:59:31.167228 10133 raft_consensus.cc:3037] T 39f1842ff1494d78bfa0811a799c4bb8 P 4b7f7ba2ac084e829c4304c16a94894f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:31.168006 10133 raft_consensus.cc:2454] T 39f1842ff1494d78bfa0811a799c4bb8 P 4b7f7ba2ac084e829c4304c16a94894f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdbecaeb8f214529b2012a9c003de2ff in term 1.
I20240502 01:59:31.168224 10208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39f1842ff1494d78bfa0811a799c4bb8" candidate_uuid: "bdbecaeb8f214529b2012a9c003de2ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e77d1c0d41034d1eb5a036dc17c071f5"
I20240502 01:59:31.168301 10022 leader_election.cc:302] T 39f1842ff1494d78bfa0811a799c4bb8 P bdbecaeb8f214529b2012a9c003de2ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b7f7ba2ac084e829c4304c16a94894f, bdbecaeb8f214529b2012a9c003de2ff; no voters:
I20240502 01:59:31.168356 10208 raft_consensus.cc:3037] T 39f1842ff1494d78bfa0811a799c4bb8 P e77d1c0d41034d1eb5a036dc17c071f5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:31.168443 10263 raft_consensus.cc:2781] T 39f1842ff1494d78bfa0811a799c4bb8 P bdbecaeb8f214529b2012a9c003de2ff [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:31.168653 10263 raft_consensus.cc:686] T 39f1842ff1494d78bfa0811a799c4bb8 P bdbecaeb8f214529b2012a9c003de2ff [term 1 LEADER]: Becoming Leader. State: Replica: bdbecaeb8f214529b2012a9c003de2ff, State: Running, Role: LEADER
I20240502 01:59:31.168852 10263 consensus_queue.cc:237] T 39f1842ff1494d78bfa0811a799c4bb8 P bdbecaeb8f214529b2012a9c003de2ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.169126 10208 raft_consensus.cc:2454] T 39f1842ff1494d78bfa0811a799c4bb8 P e77d1c0d41034d1eb5a036dc17c071f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdbecaeb8f214529b2012a9c003de2ff in term 1.
I20240502 01:59:31.169760 10263 raft_consensus.cc:482] T abf7d8b411bb46b68ac9ae77d74f8e2b P bdbecaeb8f214529b2012a9c003de2ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:31.169850 10263 raft_consensus.cc:504] T abf7d8b411bb46b68ac9ae77d74f8e2b P bdbecaeb8f214529b2012a9c003de2ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.170169 10263 leader_election.cc:288] T abf7d8b411bb46b68ac9ae77d74f8e2b P bdbecaeb8f214529b2012a9c003de2ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4b7f7ba2ac084e829c4304c16a94894f (127.6.117.194:46805), e77d1c0d41034d1eb5a036dc17c071f5 (127.6.117.195:43977)
I20240502 01:59:31.170406 10133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abf7d8b411bb46b68ac9ae77d74f8e2b" candidate_uuid: "bdbecaeb8f214529b2012a9c003de2ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b7f7ba2ac084e829c4304c16a94894f" is_pre_election: true
I20240502 01:59:31.170445 10208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abf7d8b411bb46b68ac9ae77d74f8e2b" candidate_uuid: "bdbecaeb8f214529b2012a9c003de2ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" is_pre_election: true
I20240502 01:59:31.170517 10133 raft_consensus.cc:2454] T abf7d8b411bb46b68ac9ae77d74f8e2b P 4b7f7ba2ac084e829c4304c16a94894f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bdbecaeb8f214529b2012a9c003de2ff in term 0.
I20240502 01:59:31.170526 10208 raft_consensus.cc:2454] T abf7d8b411bb46b68ac9ae77d74f8e2b P e77d1c0d41034d1eb5a036dc17c071f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bdbecaeb8f214529b2012a9c003de2ff in term 0.
I20240502 01:59:31.170596 9952 catalog_manager.cc:5484] T 39f1842ff1494d78bfa0811a799c4bb8 P bdbecaeb8f214529b2012a9c003de2ff reported cstate change: term changed from 0 to 1, leader changed from <none> to bdbecaeb8f214529b2012a9c003de2ff (127.6.117.193). New cstate: current_term: 1 leader_uuid: "bdbecaeb8f214529b2012a9c003de2ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } health_report { overall_health: UNKNOWN } } }
I20240502 01:59:31.170778 10021 leader_election.cc:302] T abf7d8b411bb46b68ac9ae77d74f8e2b P bdbecaeb8f214529b2012a9c003de2ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bdbecaeb8f214529b2012a9c003de2ff, e77d1c0d41034d1eb5a036dc17c071f5; no voters:
I20240502 01:59:31.170935 10263 raft_consensus.cc:2781] T abf7d8b411bb46b68ac9ae77d74f8e2b P bdbecaeb8f214529b2012a9c003de2ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:31.170981 10263 raft_consensus.cc:482] T abf7d8b411bb46b68ac9ae77d74f8e2b P bdbecaeb8f214529b2012a9c003de2ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:31.171013 10263 raft_consensus.cc:3037] T abf7d8b411bb46b68ac9ae77d74f8e2b P bdbecaeb8f214529b2012a9c003de2ff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:31.171756 10263 raft_consensus.cc:504] T abf7d8b411bb46b68ac9ae77d74f8e2b P bdbecaeb8f214529b2012a9c003de2ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.172062 10263 leader_election.cc:288] T abf7d8b411bb46b68ac9ae77d74f8e2b P bdbecaeb8f214529b2012a9c003de2ff [CANDIDATE]: Term 1 election: Requested vote from peers 4b7f7ba2ac084e829c4304c16a94894f (127.6.117.194:46805), e77d1c0d41034d1eb5a036dc17c071f5 (127.6.117.195:43977)
I20240502 01:59:31.172330 10133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abf7d8b411bb46b68ac9ae77d74f8e2b" candidate_uuid: "bdbecaeb8f214529b2012a9c003de2ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b7f7ba2ac084e829c4304c16a94894f"
I20240502 01:59:31.172367 10208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abf7d8b411bb46b68ac9ae77d74f8e2b" candidate_uuid: "bdbecaeb8f214529b2012a9c003de2ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e77d1c0d41034d1eb5a036dc17c071f5"
I20240502 01:59:31.172421 10133 raft_consensus.cc:3037] T abf7d8b411bb46b68ac9ae77d74f8e2b P 4b7f7ba2ac084e829c4304c16a94894f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:31.172456 10208 raft_consensus.cc:3037] T abf7d8b411bb46b68ac9ae77d74f8e2b P e77d1c0d41034d1eb5a036dc17c071f5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:31.173254 10208 raft_consensus.cc:2454] T abf7d8b411bb46b68ac9ae77d74f8e2b P e77d1c0d41034d1eb5a036dc17c071f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdbecaeb8f214529b2012a9c003de2ff in term 1.
I20240502 01:59:31.173251 10133 raft_consensus.cc:2454] T abf7d8b411bb46b68ac9ae77d74f8e2b P 4b7f7ba2ac084e829c4304c16a94894f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdbecaeb8f214529b2012a9c003de2ff in term 1.
I20240502 01:59:31.173513 10021 leader_election.cc:302] T abf7d8b411bb46b68ac9ae77d74f8e2b P bdbecaeb8f214529b2012a9c003de2ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bdbecaeb8f214529b2012a9c003de2ff, e77d1c0d41034d1eb5a036dc17c071f5; no voters:
I20240502 01:59:31.173671 10263 raft_consensus.cc:2781] T abf7d8b411bb46b68ac9ae77d74f8e2b P bdbecaeb8f214529b2012a9c003de2ff [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:31.173753 10263 raft_consensus.cc:686] T abf7d8b411bb46b68ac9ae77d74f8e2b P bdbecaeb8f214529b2012a9c003de2ff [term 1 LEADER]: Becoming Leader. State: Replica: bdbecaeb8f214529b2012a9c003de2ff, State: Running, Role: LEADER
I20240502 01:59:31.174023 10263 consensus_queue.cc:237] T abf7d8b411bb46b68ac9ae77d74f8e2b P bdbecaeb8f214529b2012a9c003de2ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.175443 9952 catalog_manager.cc:5484] T abf7d8b411bb46b68ac9ae77d74f8e2b P bdbecaeb8f214529b2012a9c003de2ff reported cstate change: term changed from 0 to 1, leader changed from <none> to bdbecaeb8f214529b2012a9c003de2ff (127.6.117.193). New cstate: current_term: 1 leader_uuid: "bdbecaeb8f214529b2012a9c003de2ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } health_report { overall_health: UNKNOWN } } }
I20240502 01:59:31.187047 10263 raft_consensus.cc:482] T e7a8fc5afcab4b1db4ebf245b66df4be P bdbecaeb8f214529b2012a9c003de2ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:31.187209 10263 raft_consensus.cc:504] T e7a8fc5afcab4b1db4ebf245b66df4be P bdbecaeb8f214529b2012a9c003de2ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.187641 10263 leader_election.cc:288] T e7a8fc5afcab4b1db4ebf245b66df4be P bdbecaeb8f214529b2012a9c003de2ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4b7f7ba2ac084e829c4304c16a94894f (127.6.117.194:46805), e77d1c0d41034d1eb5a036dc17c071f5 (127.6.117.195:43977)
I20240502 01:59:31.188064 10133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7a8fc5afcab4b1db4ebf245b66df4be" candidate_uuid: "bdbecaeb8f214529b2012a9c003de2ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b7f7ba2ac084e829c4304c16a94894f" is_pre_election: true
I20240502 01:59:31.188061 10208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7a8fc5afcab4b1db4ebf245b66df4be" candidate_uuid: "bdbecaeb8f214529b2012a9c003de2ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" is_pre_election: true
I20240502 01:59:31.188222 10133 raft_consensus.cc:2454] T e7a8fc5afcab4b1db4ebf245b66df4be P 4b7f7ba2ac084e829c4304c16a94894f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bdbecaeb8f214529b2012a9c003de2ff in term 0.
I20240502 01:59:31.188227 10208 raft_consensus.cc:2454] T e7a8fc5afcab4b1db4ebf245b66df4be P e77d1c0d41034d1eb5a036dc17c071f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bdbecaeb8f214529b2012a9c003de2ff in term 0.
I20240502 01:59:31.188529 10021 leader_election.cc:302] T e7a8fc5afcab4b1db4ebf245b66df4be P bdbecaeb8f214529b2012a9c003de2ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bdbecaeb8f214529b2012a9c003de2ff, e77d1c0d41034d1eb5a036dc17c071f5; no voters:
I20240502 01:59:31.188692 10263 raft_consensus.cc:2781] T e7a8fc5afcab4b1db4ebf245b66df4be P bdbecaeb8f214529b2012a9c003de2ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:31.188746 10263 raft_consensus.cc:482] T e7a8fc5afcab4b1db4ebf245b66df4be P bdbecaeb8f214529b2012a9c003de2ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:31.188776 10263 raft_consensus.cc:3037] T e7a8fc5afcab4b1db4ebf245b66df4be P bdbecaeb8f214529b2012a9c003de2ff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:31.189553 10263 raft_consensus.cc:504] T e7a8fc5afcab4b1db4ebf245b66df4be P bdbecaeb8f214529b2012a9c003de2ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.189868 10263 leader_election.cc:288] T e7a8fc5afcab4b1db4ebf245b66df4be P bdbecaeb8f214529b2012a9c003de2ff [CANDIDATE]: Term 1 election: Requested vote from peers 4b7f7ba2ac084e829c4304c16a94894f (127.6.117.194:46805), e77d1c0d41034d1eb5a036dc17c071f5 (127.6.117.195:43977)
I20240502 01:59:31.190237 10133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7a8fc5afcab4b1db4ebf245b66df4be" candidate_uuid: "bdbecaeb8f214529b2012a9c003de2ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b7f7ba2ac084e829c4304c16a94894f"
I20240502 01:59:31.190237 10208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7a8fc5afcab4b1db4ebf245b66df4be" candidate_uuid: "bdbecaeb8f214529b2012a9c003de2ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e77d1c0d41034d1eb5a036dc17c071f5"
I20240502 01:59:31.190419 10133 raft_consensus.cc:3037] T e7a8fc5afcab4b1db4ebf245b66df4be P 4b7f7ba2ac084e829c4304c16a94894f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:31.190418 10208 raft_consensus.cc:3037] T e7a8fc5afcab4b1db4ebf245b66df4be P e77d1c0d41034d1eb5a036dc17c071f5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:31.191370 10208 raft_consensus.cc:2454] T e7a8fc5afcab4b1db4ebf245b66df4be P e77d1c0d41034d1eb5a036dc17c071f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdbecaeb8f214529b2012a9c003de2ff in term 1.
I20240502 01:59:31.191365 10133 raft_consensus.cc:2454] T e7a8fc5afcab4b1db4ebf245b66df4be P 4b7f7ba2ac084e829c4304c16a94894f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdbecaeb8f214529b2012a9c003de2ff in term 1.
I20240502 01:59:31.191654 10021 leader_election.cc:302] T e7a8fc5afcab4b1db4ebf245b66df4be P bdbecaeb8f214529b2012a9c003de2ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bdbecaeb8f214529b2012a9c003de2ff, e77d1c0d41034d1eb5a036dc17c071f5; no voters:
I20240502 01:59:31.191800 10263 raft_consensus.cc:2781] T e7a8fc5afcab4b1db4ebf245b66df4be P bdbecaeb8f214529b2012a9c003de2ff [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:31.191902 10263 raft_consensus.cc:686] T e7a8fc5afcab4b1db4ebf245b66df4be P bdbecaeb8f214529b2012a9c003de2ff [term 1 LEADER]: Becoming Leader. State: Replica: bdbecaeb8f214529b2012a9c003de2ff, State: Running, Role: LEADER
I20240502 01:59:31.192032 10263 consensus_queue.cc:237] T e7a8fc5afcab4b1db4ebf245b66df4be P bdbecaeb8f214529b2012a9c003de2ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.193785 9952 catalog_manager.cc:5484] T e7a8fc5afcab4b1db4ebf245b66df4be P bdbecaeb8f214529b2012a9c003de2ff reported cstate change: term changed from 0 to 1, leader changed from <none> to bdbecaeb8f214529b2012a9c003de2ff (127.6.117.193). New cstate: current_term: 1 leader_uuid: "bdbecaeb8f214529b2012a9c003de2ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } health_report { overall_health: UNKNOWN } } }
I20240502 01:59:31.195567 10258 raft_consensus.cc:482] T 73ea0bfb40ad43a68a6289ff1b35d005 P 4b7f7ba2ac084e829c4304c16a94894f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:31.195662 10258 raft_consensus.cc:504] T 73ea0bfb40ad43a68a6289ff1b35d005 P 4b7f7ba2ac084e829c4304c16a94894f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.196063 10258 leader_election.cc:288] T 73ea0bfb40ad43a68a6289ff1b35d005 P 4b7f7ba2ac084e829c4304c16a94894f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bdbecaeb8f214529b2012a9c003de2ff (127.6.117.193:34597), e77d1c0d41034d1eb5a036dc17c071f5 (127.6.117.195:43977)
I20240502 01:59:31.199368 10057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73ea0bfb40ad43a68a6289ff1b35d005" candidate_uuid: "4b7f7ba2ac084e829c4304c16a94894f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdbecaeb8f214529b2012a9c003de2ff" is_pre_election: true
I20240502 01:59:31.199515 10057 raft_consensus.cc:2454] T 73ea0bfb40ad43a68a6289ff1b35d005 P bdbecaeb8f214529b2012a9c003de2ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b7f7ba2ac084e829c4304c16a94894f in term 0.
I20240502 01:59:31.199689 10208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73ea0bfb40ad43a68a6289ff1b35d005" candidate_uuid: "4b7f7ba2ac084e829c4304c16a94894f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" is_pre_election: true
I20240502 01:59:31.199826 10099 leader_election.cc:302] T 73ea0bfb40ad43a68a6289ff1b35d005 P 4b7f7ba2ac084e829c4304c16a94894f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b7f7ba2ac084e829c4304c16a94894f, bdbecaeb8f214529b2012a9c003de2ff; no voters:
I20240502 01:59:31.199810 10208 raft_consensus.cc:2454] T 73ea0bfb40ad43a68a6289ff1b35d005 P e77d1c0d41034d1eb5a036dc17c071f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b7f7ba2ac084e829c4304c16a94894f in term 0.
I20240502 01:59:31.200014 10258 raft_consensus.cc:2781] T 73ea0bfb40ad43a68a6289ff1b35d005 P 4b7f7ba2ac084e829c4304c16a94894f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:31.200076 10258 raft_consensus.cc:482] T 73ea0bfb40ad43a68a6289ff1b35d005 P 4b7f7ba2ac084e829c4304c16a94894f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:31.200115 10258 raft_consensus.cc:3037] T 73ea0bfb40ad43a68a6289ff1b35d005 P 4b7f7ba2ac084e829c4304c16a94894f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:31.200948 10258 raft_consensus.cc:504] T 73ea0bfb40ad43a68a6289ff1b35d005 P 4b7f7ba2ac084e829c4304c16a94894f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.201257 10258 leader_election.cc:288] T 73ea0bfb40ad43a68a6289ff1b35d005 P 4b7f7ba2ac084e829c4304c16a94894f [CANDIDATE]: Term 1 election: Requested vote from peers bdbecaeb8f214529b2012a9c003de2ff (127.6.117.193:34597), e77d1c0d41034d1eb5a036dc17c071f5 (127.6.117.195:43977)
I20240502 01:59:31.201540 10208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73ea0bfb40ad43a68a6289ff1b35d005" candidate_uuid: "4b7f7ba2ac084e829c4304c16a94894f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e77d1c0d41034d1eb5a036dc17c071f5"
I20240502 01:59:31.201634 10208 raft_consensus.cc:3037] T 73ea0bfb40ad43a68a6289ff1b35d005 P e77d1c0d41034d1eb5a036dc17c071f5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:31.202401 10208 raft_consensus.cc:2454] T 73ea0bfb40ad43a68a6289ff1b35d005 P e77d1c0d41034d1eb5a036dc17c071f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b7f7ba2ac084e829c4304c16a94894f in term 1.
I20240502 01:59:31.202594 10057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73ea0bfb40ad43a68a6289ff1b35d005" candidate_uuid: "4b7f7ba2ac084e829c4304c16a94894f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdbecaeb8f214529b2012a9c003de2ff"
I20240502 01:59:31.202670 10057 raft_consensus.cc:3037] T 73ea0bfb40ad43a68a6289ff1b35d005 P bdbecaeb8f214529b2012a9c003de2ff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:31.202647 10097 leader_election.cc:302] T 73ea0bfb40ad43a68a6289ff1b35d005 P 4b7f7ba2ac084e829c4304c16a94894f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b7f7ba2ac084e829c4304c16a94894f, e77d1c0d41034d1eb5a036dc17c071f5; no voters:
I20240502 01:59:31.202847 10258 raft_consensus.cc:2781] T 73ea0bfb40ad43a68a6289ff1b35d005 P 4b7f7ba2ac084e829c4304c16a94894f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:31.203068 10258 raft_consensus.cc:686] T 73ea0bfb40ad43a68a6289ff1b35d005 P 4b7f7ba2ac084e829c4304c16a94894f [term 1 LEADER]: Becoming Leader. State: Replica: 4b7f7ba2ac084e829c4304c16a94894f, State: Running, Role: LEADER
I20240502 01:59:31.203212 10258 consensus_queue.cc:237] T 73ea0bfb40ad43a68a6289ff1b35d005 P 4b7f7ba2ac084e829c4304c16a94894f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.203526 10057 raft_consensus.cc:2454] T 73ea0bfb40ad43a68a6289ff1b35d005 P bdbecaeb8f214529b2012a9c003de2ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b7f7ba2ac084e829c4304c16a94894f in term 1.
I20240502 01:59:31.204957 9952 catalog_manager.cc:5484] T 73ea0bfb40ad43a68a6289ff1b35d005 P 4b7f7ba2ac084e829c4304c16a94894f reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b7f7ba2ac084e829c4304c16a94894f (127.6.117.194). New cstate: current_term: 1 leader_uuid: "4b7f7ba2ac084e829c4304c16a94894f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } health_report { overall_health: UNKNOWN } } }
I20240502 01:59:31.212358 10263 raft_consensus.cc:482] T 66ccad5a808b42378ccecc9da81cee74 P bdbecaeb8f214529b2012a9c003de2ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:31.212529 10263 raft_consensus.cc:504] T 66ccad5a808b42378ccecc9da81cee74 P bdbecaeb8f214529b2012a9c003de2ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.212951 10263 leader_election.cc:288] T 66ccad5a808b42378ccecc9da81cee74 P bdbecaeb8f214529b2012a9c003de2ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4b7f7ba2ac084e829c4304c16a94894f (127.6.117.194:46805), e77d1c0d41034d1eb5a036dc17c071f5 (127.6.117.195:43977)
I20240502 01:59:31.213332 10133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66ccad5a808b42378ccecc9da81cee74" candidate_uuid: "bdbecaeb8f214529b2012a9c003de2ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b7f7ba2ac084e829c4304c16a94894f" is_pre_election: true
I20240502 01:59:31.213339 10208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66ccad5a808b42378ccecc9da81cee74" candidate_uuid: "bdbecaeb8f214529b2012a9c003de2ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" is_pre_election: true
I20240502 01:59:31.213480 10133 raft_consensus.cc:2454] T 66ccad5a808b42378ccecc9da81cee74 P 4b7f7ba2ac084e829c4304c16a94894f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bdbecaeb8f214529b2012a9c003de2ff in term 0.
I20240502 01:59:31.213487 10208 raft_consensus.cc:2454] T 66ccad5a808b42378ccecc9da81cee74 P e77d1c0d41034d1eb5a036dc17c071f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bdbecaeb8f214529b2012a9c003de2ff in term 0.
I20240502 01:59:31.213733 10021 leader_election.cc:302] T 66ccad5a808b42378ccecc9da81cee74 P bdbecaeb8f214529b2012a9c003de2ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bdbecaeb8f214529b2012a9c003de2ff, e77d1c0d41034d1eb5a036dc17c071f5; no voters:
I20240502 01:59:31.213886 10263 raft_consensus.cc:2781] T 66ccad5a808b42378ccecc9da81cee74 P bdbecaeb8f214529b2012a9c003de2ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:31.213965 10263 raft_consensus.cc:482] T 66ccad5a808b42378ccecc9da81cee74 P bdbecaeb8f214529b2012a9c003de2ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:31.213994 10263 raft_consensus.cc:3037] T 66ccad5a808b42378ccecc9da81cee74 P bdbecaeb8f214529b2012a9c003de2ff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:31.214758 10263 raft_consensus.cc:504] T 66ccad5a808b42378ccecc9da81cee74 P bdbecaeb8f214529b2012a9c003de2ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.215072 10263 leader_election.cc:288] T 66ccad5a808b42378ccecc9da81cee74 P bdbecaeb8f214529b2012a9c003de2ff [CANDIDATE]: Term 1 election: Requested vote from peers 4b7f7ba2ac084e829c4304c16a94894f (127.6.117.194:46805), e77d1c0d41034d1eb5a036dc17c071f5 (127.6.117.195:43977)
I20240502 01:59:31.215358 10133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66ccad5a808b42378ccecc9da81cee74" candidate_uuid: "bdbecaeb8f214529b2012a9c003de2ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b7f7ba2ac084e829c4304c16a94894f"
I20240502 01:59:31.215368 10208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66ccad5a808b42378ccecc9da81cee74" candidate_uuid: "bdbecaeb8f214529b2012a9c003de2ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e77d1c0d41034d1eb5a036dc17c071f5"
I20240502 01:59:31.215514 10208 raft_consensus.cc:3037] T 66ccad5a808b42378ccecc9da81cee74 P e77d1c0d41034d1eb5a036dc17c071f5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:31.215581 10133 raft_consensus.cc:3037] T 66ccad5a808b42378ccecc9da81cee74 P 4b7f7ba2ac084e829c4304c16a94894f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:31.216248 10208 raft_consensus.cc:2454] T 66ccad5a808b42378ccecc9da81cee74 P e77d1c0d41034d1eb5a036dc17c071f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdbecaeb8f214529b2012a9c003de2ff in term 1.
I20240502 01:59:31.216318 10133 raft_consensus.cc:2454] T 66ccad5a808b42378ccecc9da81cee74 P 4b7f7ba2ac084e829c4304c16a94894f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdbecaeb8f214529b2012a9c003de2ff in term 1.
I20240502 01:59:31.216529 10021 leader_election.cc:302] T 66ccad5a808b42378ccecc9da81cee74 P bdbecaeb8f214529b2012a9c003de2ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bdbecaeb8f214529b2012a9c003de2ff, e77d1c0d41034d1eb5a036dc17c071f5; no voters:
I20240502 01:59:31.216686 10263 raft_consensus.cc:2781] T 66ccad5a808b42378ccecc9da81cee74 P bdbecaeb8f214529b2012a9c003de2ff [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:31.216768 10263 raft_consensus.cc:686] T 66ccad5a808b42378ccecc9da81cee74 P bdbecaeb8f214529b2012a9c003de2ff [term 1 LEADER]: Becoming Leader. State: Replica: bdbecaeb8f214529b2012a9c003de2ff, State: Running, Role: LEADER
I20240502 01:59:31.216902 10263 consensus_queue.cc:237] T 66ccad5a808b42378ccecc9da81cee74 P bdbecaeb8f214529b2012a9c003de2ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.218420 9952 catalog_manager.cc:5484] T 66ccad5a808b42378ccecc9da81cee74 P bdbecaeb8f214529b2012a9c003de2ff reported cstate change: term changed from 0 to 1, leader changed from <none> to bdbecaeb8f214529b2012a9c003de2ff (127.6.117.193). New cstate: current_term: 1 leader_uuid: "bdbecaeb8f214529b2012a9c003de2ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } health_report { overall_health: UNKNOWN } } }
I20240502 01:59:31.222080 10258 raft_consensus.cc:482] T 19ff9c216a094fde86ea9f9648bf08cb P 4b7f7ba2ac084e829c4304c16a94894f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:31.222199 10258 raft_consensus.cc:504] T 19ff9c216a094fde86ea9f9648bf08cb P 4b7f7ba2ac084e829c4304c16a94894f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.222587 10258 leader_election.cc:288] T 19ff9c216a094fde86ea9f9648bf08cb P 4b7f7ba2ac084e829c4304c16a94894f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bdbecaeb8f214529b2012a9c003de2ff (127.6.117.193:34597), e77d1c0d41034d1eb5a036dc17c071f5 (127.6.117.195:43977)
I20240502 01:59:31.223011 10057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19ff9c216a094fde86ea9f9648bf08cb" candidate_uuid: "4b7f7ba2ac084e829c4304c16a94894f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdbecaeb8f214529b2012a9c003de2ff" is_pre_election: true
I20240502 01:59:31.223167 10258 raft_consensus.cc:482] T 1851ea6467584b0ea32a1f68696803d0 P 4b7f7ba2ac084e829c4304c16a94894f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:31.223206 10057 raft_consensus.cc:2454] T 19ff9c216a094fde86ea9f9648bf08cb P bdbecaeb8f214529b2012a9c003de2ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b7f7ba2ac084e829c4304c16a94894f in term 0.
I20240502 01:59:31.223261 10258 raft_consensus.cc:504] T 1851ea6467584b0ea32a1f68696803d0 P 4b7f7ba2ac084e829c4304c16a94894f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.223489 10099 leader_election.cc:302] T 19ff9c216a094fde86ea9f9648bf08cb P 4b7f7ba2ac084e829c4304c16a94894f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b7f7ba2ac084e829c4304c16a94894f, bdbecaeb8f214529b2012a9c003de2ff; no voters:
I20240502 01:59:31.223572 10258 leader_election.cc:288] T 1851ea6467584b0ea32a1f68696803d0 P 4b7f7ba2ac084e829c4304c16a94894f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bdbecaeb8f214529b2012a9c003de2ff (127.6.117.193:34597), e77d1c0d41034d1eb5a036dc17c071f5 (127.6.117.195:43977)
I20240502 01:59:31.223666 10258 raft_consensus.cc:2781] T 19ff9c216a094fde86ea9f9648bf08cb P 4b7f7ba2ac084e829c4304c16a94894f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:31.223011 10208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19ff9c216a094fde86ea9f9648bf08cb" candidate_uuid: "4b7f7ba2ac084e829c4304c16a94894f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" is_pre_election: true
I20240502 01:59:31.223707 10258 raft_consensus.cc:482] T 19ff9c216a094fde86ea9f9648bf08cb P 4b7f7ba2ac084e829c4304c16a94894f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:31.223738 10258 raft_consensus.cc:3037] T 19ff9c216a094fde86ea9f9648bf08cb P 4b7f7ba2ac084e829c4304c16a94894f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:31.223763 10208 raft_consensus.cc:2454] T 19ff9c216a094fde86ea9f9648bf08cb P e77d1c0d41034d1eb5a036dc17c071f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b7f7ba2ac084e829c4304c16a94894f in term 0.
I20240502 01:59:31.224023 10057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1851ea6467584b0ea32a1f68696803d0" candidate_uuid: "4b7f7ba2ac084e829c4304c16a94894f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdbecaeb8f214529b2012a9c003de2ff" is_pre_election: true
I20240502 01:59:31.224119 10057 raft_consensus.cc:2454] T 1851ea6467584b0ea32a1f68696803d0 P bdbecaeb8f214529b2012a9c003de2ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b7f7ba2ac084e829c4304c16a94894f in term 0.
I20240502 01:59:31.224385 10099 leader_election.cc:302] T 1851ea6467584b0ea32a1f68696803d0 P 4b7f7ba2ac084e829c4304c16a94894f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b7f7ba2ac084e829c4304c16a94894f, bdbecaeb8f214529b2012a9c003de2ff; no voters:
I20240502 01:59:31.224546 10304 raft_consensus.cc:2781] T 1851ea6467584b0ea32a1f68696803d0 P 4b7f7ba2ac084e829c4304c16a94894f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:31.224586 10304 raft_consensus.cc:482] T 1851ea6467584b0ea32a1f68696803d0 P 4b7f7ba2ac084e829c4304c16a94894f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:31.224627 10304 raft_consensus.cc:3037] T 1851ea6467584b0ea32a1f68696803d0 P 4b7f7ba2ac084e829c4304c16a94894f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:31.224800 10258 raft_consensus.cc:504] T 19ff9c216a094fde86ea9f9648bf08cb P 4b7f7ba2ac084e829c4304c16a94894f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.225128 10208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1851ea6467584b0ea32a1f68696803d0" candidate_uuid: "4b7f7ba2ac084e829c4304c16a94894f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" is_pre_election: true
I20240502 01:59:31.225226 10208 raft_consensus.cc:2454] T 1851ea6467584b0ea32a1f68696803d0 P e77d1c0d41034d1eb5a036dc17c071f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b7f7ba2ac084e829c4304c16a94894f in term 0.
I20240502 01:59:31.225323 10057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19ff9c216a094fde86ea9f9648bf08cb" candidate_uuid: "4b7f7ba2ac084e829c4304c16a94894f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdbecaeb8f214529b2012a9c003de2ff"
I20240502 01:59:31.225406 10057 raft_consensus.cc:3037] T 19ff9c216a094fde86ea9f9648bf08cb P bdbecaeb8f214529b2012a9c003de2ff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:31.225638 10304 raft_consensus.cc:504] T 1851ea6467584b0ea32a1f68696803d0 P 4b7f7ba2ac084e829c4304c16a94894f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.225987 10304 leader_election.cc:288] T 1851ea6467584b0ea32a1f68696803d0 P 4b7f7ba2ac084e829c4304c16a94894f [CANDIDATE]: Term 1 election: Requested vote from peers bdbecaeb8f214529b2012a9c003de2ff (127.6.117.193:34597), e77d1c0d41034d1eb5a036dc17c071f5 (127.6.117.195:43977)
I20240502 01:59:31.226032 10057 raft_consensus.cc:2454] T 19ff9c216a094fde86ea9f9648bf08cb P bdbecaeb8f214529b2012a9c003de2ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b7f7ba2ac084e829c4304c16a94894f in term 1.
I20240502 01:59:31.226248 10099 leader_election.cc:302] T 19ff9c216a094fde86ea9f9648bf08cb P 4b7f7ba2ac084e829c4304c16a94894f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b7f7ba2ac084e829c4304c16a94894f, bdbecaeb8f214529b2012a9c003de2ff; no voters:
I20240502 01:59:31.226274 10208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1851ea6467584b0ea32a1f68696803d0" candidate_uuid: "4b7f7ba2ac084e829c4304c16a94894f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e77d1c0d41034d1eb5a036dc17c071f5"
I20240502 01:59:31.226312 10057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1851ea6467584b0ea32a1f68696803d0" candidate_uuid: "4b7f7ba2ac084e829c4304c16a94894f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdbecaeb8f214529b2012a9c003de2ff"
I20240502 01:59:31.226370 10208 raft_consensus.cc:3037] T 1851ea6467584b0ea32a1f68696803d0 P e77d1c0d41034d1eb5a036dc17c071f5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:31.226401 10304 raft_consensus.cc:2781] T 19ff9c216a094fde86ea9f9648bf08cb P 4b7f7ba2ac084e829c4304c16a94894f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:31.226636 10304 raft_consensus.cc:686] T 19ff9c216a094fde86ea9f9648bf08cb P 4b7f7ba2ac084e829c4304c16a94894f [term 1 LEADER]: Becoming Leader. State: Replica: 4b7f7ba2ac084e829c4304c16a94894f, State: Running, Role: LEADER
I20240502 01:59:31.226770 10207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19ff9c216a094fde86ea9f9648bf08cb" candidate_uuid: "4b7f7ba2ac084e829c4304c16a94894f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e77d1c0d41034d1eb5a036dc17c071f5"
I20240502 01:59:31.226892 10207 raft_consensus.cc:3037] T 19ff9c216a094fde86ea9f9648bf08cb P e77d1c0d41034d1eb5a036dc17c071f5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:31.226789 10304 consensus_queue.cc:237] T 19ff9c216a094fde86ea9f9648bf08cb P 4b7f7ba2ac084e829c4304c16a94894f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.227552 10208 raft_consensus.cc:2454] T 1851ea6467584b0ea32a1f68696803d0 P e77d1c0d41034d1eb5a036dc17c071f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b7f7ba2ac084e829c4304c16a94894f in term 1.
I20240502 01:59:31.227775 10097 leader_election.cc:302] T 1851ea6467584b0ea32a1f68696803d0 P 4b7f7ba2ac084e829c4304c16a94894f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b7f7ba2ac084e829c4304c16a94894f, e77d1c0d41034d1eb5a036dc17c071f5; no voters:
I20240502 01:59:31.227902 10304 raft_consensus.cc:2781] T 1851ea6467584b0ea32a1f68696803d0 P 4b7f7ba2ac084e829c4304c16a94894f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:31.227959 10304 raft_consensus.cc:686] T 1851ea6467584b0ea32a1f68696803d0 P 4b7f7ba2ac084e829c4304c16a94894f [term 1 LEADER]: Becoming Leader. State: Replica: 4b7f7ba2ac084e829c4304c16a94894f, State: Running, Role: LEADER
I20240502 01:59:31.228062 10304 consensus_queue.cc:237] T 1851ea6467584b0ea32a1f68696803d0 P 4b7f7ba2ac084e829c4304c16a94894f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.229251 9952 catalog_manager.cc:5484] T 19ff9c216a094fde86ea9f9648bf08cb P 4b7f7ba2ac084e829c4304c16a94894f reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b7f7ba2ac084e829c4304c16a94894f (127.6.117.194). New cstate: current_term: 1 leader_uuid: "4b7f7ba2ac084e829c4304c16a94894f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } health_report { overall_health: UNKNOWN } } }
I20240502 01:59:31.229583 10207 raft_consensus.cc:2454] T 19ff9c216a094fde86ea9f9648bf08cb P e77d1c0d41034d1eb5a036dc17c071f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b7f7ba2ac084e829c4304c16a94894f in term 1.
I20240502 01:59:31.226375 10057 raft_consensus.cc:3037] T 1851ea6467584b0ea32a1f68696803d0 P bdbecaeb8f214529b2012a9c003de2ff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:31.230324 10057 raft_consensus.cc:2454] T 1851ea6467584b0ea32a1f68696803d0 P bdbecaeb8f214529b2012a9c003de2ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b7f7ba2ac084e829c4304c16a94894f in term 1.
I20240502 01:59:31.230973 10258 leader_election.cc:288] T 19ff9c216a094fde86ea9f9648bf08cb P 4b7f7ba2ac084e829c4304c16a94894f [CANDIDATE]: Term 1 election: Requested vote from peers bdbecaeb8f214529b2012a9c003de2ff (127.6.117.193:34597), e77d1c0d41034d1eb5a036dc17c071f5 (127.6.117.195:43977)
I20240502 01:59:31.232894 9952 catalog_manager.cc:5484] T 1851ea6467584b0ea32a1f68696803d0 P 4b7f7ba2ac084e829c4304c16a94894f reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b7f7ba2ac084e829c4304c16a94894f (127.6.117.194). New cstate: current_term: 1 leader_uuid: "4b7f7ba2ac084e829c4304c16a94894f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } health_report { overall_health: UNKNOWN } } }
I20240502 01:59:31.243422 10264 raft_consensus.cc:482] T b4409766ce994f698c31aede604a6927 P e77d1c0d41034d1eb5a036dc17c071f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:31.243588 10264 raft_consensus.cc:504] T b4409766ce994f698c31aede604a6927 P e77d1c0d41034d1eb5a036dc17c071f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.244086 10264 leader_election.cc:288] T b4409766ce994f698c31aede604a6927 P e77d1c0d41034d1eb5a036dc17c071f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bdbecaeb8f214529b2012a9c003de2ff (127.6.117.193:34597), 4b7f7ba2ac084e829c4304c16a94894f (127.6.117.194:46805)
I20240502 01:59:31.247484 10057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4409766ce994f698c31aede604a6927" candidate_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdbecaeb8f214529b2012a9c003de2ff" is_pre_election: true
I20240502 01:59:31.247484 10133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4409766ce994f698c31aede604a6927" candidate_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b7f7ba2ac084e829c4304c16a94894f" is_pre_election: true
I20240502 01:59:31.247648 10057 raft_consensus.cc:2454] T b4409766ce994f698c31aede604a6927 P bdbecaeb8f214529b2012a9c003de2ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e77d1c0d41034d1eb5a036dc17c071f5 in term 0.
I20240502 01:59:31.247690 10133 raft_consensus.cc:2454] T b4409766ce994f698c31aede604a6927 P 4b7f7ba2ac084e829c4304c16a94894f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e77d1c0d41034d1eb5a036dc17c071f5 in term 0.
I20240502 01:59:31.247928 10174 leader_election.cc:302] T b4409766ce994f698c31aede604a6927 P e77d1c0d41034d1eb5a036dc17c071f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4b7f7ba2ac084e829c4304c16a94894f, bdbecaeb8f214529b2012a9c003de2ff, e77d1c0d41034d1eb5a036dc17c071f5; no voters:
I20240502 01:59:31.248121 10264 raft_consensus.cc:2781] T b4409766ce994f698c31aede604a6927 P e77d1c0d41034d1eb5a036dc17c071f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:31.248176 10264 raft_consensus.cc:482] T b4409766ce994f698c31aede604a6927 P e77d1c0d41034d1eb5a036dc17c071f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:31.248203 10264 raft_consensus.cc:3037] T b4409766ce994f698c31aede604a6927 P e77d1c0d41034d1eb5a036dc17c071f5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:31.249043 10264 raft_consensus.cc:504] T b4409766ce994f698c31aede604a6927 P e77d1c0d41034d1eb5a036dc17c071f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.249380 10264 leader_election.cc:288] T b4409766ce994f698c31aede604a6927 P e77d1c0d41034d1eb5a036dc17c071f5 [CANDIDATE]: Term 1 election: Requested vote from peers bdbecaeb8f214529b2012a9c003de2ff (127.6.117.193:34597), 4b7f7ba2ac084e829c4304c16a94894f (127.6.117.194:46805)
I20240502 01:59:31.249714 10057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4409766ce994f698c31aede604a6927" candidate_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdbecaeb8f214529b2012a9c003de2ff"
I20240502 01:59:31.249748 10133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4409766ce994f698c31aede604a6927" candidate_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b7f7ba2ac084e829c4304c16a94894f"
I20240502 01:59:31.249809 10057 raft_consensus.cc:3037] T b4409766ce994f698c31aede604a6927 P bdbecaeb8f214529b2012a9c003de2ff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:31.249845 10133 raft_consensus.cc:3037] T b4409766ce994f698c31aede604a6927 P 4b7f7ba2ac084e829c4304c16a94894f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:31.250501 10057 raft_consensus.cc:2454] T b4409766ce994f698c31aede604a6927 P bdbecaeb8f214529b2012a9c003de2ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e77d1c0d41034d1eb5a036dc17c071f5 in term 1.
I20240502 01:59:31.250581 10133 raft_consensus.cc:2454] T b4409766ce994f698c31aede604a6927 P 4b7f7ba2ac084e829c4304c16a94894f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e77d1c0d41034d1eb5a036dc17c071f5 in term 1.
I20240502 01:59:31.250794 10174 leader_election.cc:302] T b4409766ce994f698c31aede604a6927 P e77d1c0d41034d1eb5a036dc17c071f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bdbecaeb8f214529b2012a9c003de2ff, e77d1c0d41034d1eb5a036dc17c071f5; no voters:
I20240502 01:59:31.250963 10264 raft_consensus.cc:2781] T b4409766ce994f698c31aede604a6927 P e77d1c0d41034d1eb5a036dc17c071f5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:31.251228 10264 raft_consensus.cc:686] T b4409766ce994f698c31aede604a6927 P e77d1c0d41034d1eb5a036dc17c071f5 [term 1 LEADER]: Becoming Leader. State: Replica: e77d1c0d41034d1eb5a036dc17c071f5, State: Running, Role: LEADER
I20240502 01:59:31.251363 10264 consensus_queue.cc:237] T b4409766ce994f698c31aede604a6927 P e77d1c0d41034d1eb5a036dc17c071f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.251900 10258 raft_consensus.cc:482] T bceb844e50b84e278c0c49f4f4912e00 P 4b7f7ba2ac084e829c4304c16a94894f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:31.251986 10258 raft_consensus.cc:504] T bceb844e50b84e278c0c49f4f4912e00 P 4b7f7ba2ac084e829c4304c16a94894f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.252530 10258 leader_election.cc:288] T bceb844e50b84e278c0c49f4f4912e00 P 4b7f7ba2ac084e829c4304c16a94894f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bdbecaeb8f214529b2012a9c003de2ff (127.6.117.193:34597), e77d1c0d41034d1eb5a036dc17c071f5 (127.6.117.195:43977)
I20240502 01:59:31.252624 10057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bceb844e50b84e278c0c49f4f4912e00" candidate_uuid: "4b7f7ba2ac084e829c4304c16a94894f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdbecaeb8f214529b2012a9c003de2ff" is_pre_election: true
I20240502 01:59:31.252719 10057 raft_consensus.cc:2454] T bceb844e50b84e278c0c49f4f4912e00 P bdbecaeb8f214529b2012a9c003de2ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b7f7ba2ac084e829c4304c16a94894f in term 0.
I20240502 01:59:31.252846 10207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bceb844e50b84e278c0c49f4f4912e00" candidate_uuid: "4b7f7ba2ac084e829c4304c16a94894f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" is_pre_election: true
I20240502 01:59:31.252945 10207 raft_consensus.cc:2454] T bceb844e50b84e278c0c49f4f4912e00 P e77d1c0d41034d1eb5a036dc17c071f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b7f7ba2ac084e829c4304c16a94894f in term 0.
I20240502 01:59:31.252981 10099 leader_election.cc:302] T bceb844e50b84e278c0c49f4f4912e00 P 4b7f7ba2ac084e829c4304c16a94894f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b7f7ba2ac084e829c4304c16a94894f, bdbecaeb8f214529b2012a9c003de2ff; no voters:
I20240502 01:59:31.253054 9952 catalog_manager.cc:5484] T b4409766ce994f698c31aede604a6927 P e77d1c0d41034d1eb5a036dc17c071f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to e77d1c0d41034d1eb5a036dc17c071f5 (127.6.117.195). New cstate: current_term: 1 leader_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } health_report { overall_health: HEALTHY } } }
I20240502 01:59:31.253262 10258 raft_consensus.cc:2781] T bceb844e50b84e278c0c49f4f4912e00 P 4b7f7ba2ac084e829c4304c16a94894f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:31.253317 10258 raft_consensus.cc:482] T bceb844e50b84e278c0c49f4f4912e00 P 4b7f7ba2ac084e829c4304c16a94894f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:31.253361 10258 raft_consensus.cc:3037] T bceb844e50b84e278c0c49f4f4912e00 P 4b7f7ba2ac084e829c4304c16a94894f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:31.254204 10258 raft_consensus.cc:504] T bceb844e50b84e278c0c49f4f4912e00 P 4b7f7ba2ac084e829c4304c16a94894f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.254487 10258 leader_election.cc:288] T bceb844e50b84e278c0c49f4f4912e00 P 4b7f7ba2ac084e829c4304c16a94894f [CANDIDATE]: Term 1 election: Requested vote from peers bdbecaeb8f214529b2012a9c003de2ff (127.6.117.193:34597), e77d1c0d41034d1eb5a036dc17c071f5 (127.6.117.195:43977)
I20240502 01:59:31.254745 10057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bceb844e50b84e278c0c49f4f4912e00" candidate_uuid: "4b7f7ba2ac084e829c4304c16a94894f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdbecaeb8f214529b2012a9c003de2ff"
I20240502 01:59:31.254745 10207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bceb844e50b84e278c0c49f4f4912e00" candidate_uuid: "4b7f7ba2ac084e829c4304c16a94894f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e77d1c0d41034d1eb5a036dc17c071f5"
I20240502 01:59:31.254873 10207 raft_consensus.cc:3037] T bceb844e50b84e278c0c49f4f4912e00 P e77d1c0d41034d1eb5a036dc17c071f5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:31.254873 10057 raft_consensus.cc:3037] T bceb844e50b84e278c0c49f4f4912e00 P bdbecaeb8f214529b2012a9c003de2ff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:31.255686 10207 raft_consensus.cc:2454] T bceb844e50b84e278c0c49f4f4912e00 P e77d1c0d41034d1eb5a036dc17c071f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b7f7ba2ac084e829c4304c16a94894f in term 1.
I20240502 01:59:31.255686 10057 raft_consensus.cc:2454] T bceb844e50b84e278c0c49f4f4912e00 P bdbecaeb8f214529b2012a9c003de2ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b7f7ba2ac084e829c4304c16a94894f in term 1.
I20240502 01:59:31.255970 10097 leader_election.cc:302] T bceb844e50b84e278c0c49f4f4912e00 P 4b7f7ba2ac084e829c4304c16a94894f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b7f7ba2ac084e829c4304c16a94894f, e77d1c0d41034d1eb5a036dc17c071f5; no voters:
I20240502 01:59:31.256126 10258 raft_consensus.cc:2781] T bceb844e50b84e278c0c49f4f4912e00 P 4b7f7ba2ac084e829c4304c16a94894f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:31.256223 10258 raft_consensus.cc:686] T bceb844e50b84e278c0c49f4f4912e00 P 4b7f7ba2ac084e829c4304c16a94894f [term 1 LEADER]: Becoming Leader. State: Replica: 4b7f7ba2ac084e829c4304c16a94894f, State: Running, Role: LEADER
I20240502 01:59:31.256385 10258 consensus_queue.cc:237] T bceb844e50b84e278c0c49f4f4912e00 P 4b7f7ba2ac084e829c4304c16a94894f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.257838 9952 catalog_manager.cc:5484] T bceb844e50b84e278c0c49f4f4912e00 P 4b7f7ba2ac084e829c4304c16a94894f reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b7f7ba2ac084e829c4304c16a94894f (127.6.117.194). New cstate: current_term: 1 leader_uuid: "4b7f7ba2ac084e829c4304c16a94894f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } health_report { overall_health: UNKNOWN } } }
I20240502 01:59:31.304388 10263 raft_consensus.cc:482] T 4885222f238c4d8ea20ebc4c7e171694 P bdbecaeb8f214529b2012a9c003de2ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:31.304544 10263 raft_consensus.cc:504] T 4885222f238c4d8ea20ebc4c7e171694 P bdbecaeb8f214529b2012a9c003de2ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.305019 10263 leader_election.cc:288] T 4885222f238c4d8ea20ebc4c7e171694 P bdbecaeb8f214529b2012a9c003de2ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4b7f7ba2ac084e829c4304c16a94894f (127.6.117.194:46805), e77d1c0d41034d1eb5a036dc17c071f5 (127.6.117.195:43977)
I20240502 01:59:31.305433 10133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4885222f238c4d8ea20ebc4c7e171694" candidate_uuid: "bdbecaeb8f214529b2012a9c003de2ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b7f7ba2ac084e829c4304c16a94894f" is_pre_election: true
I20240502 01:59:31.305441 10207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4885222f238c4d8ea20ebc4c7e171694" candidate_uuid: "bdbecaeb8f214529b2012a9c003de2ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" is_pre_election: true
I20240502 01:59:31.305574 10133 raft_consensus.cc:2454] T 4885222f238c4d8ea20ebc4c7e171694 P 4b7f7ba2ac084e829c4304c16a94894f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bdbecaeb8f214529b2012a9c003de2ff in term 0.
I20240502 01:59:31.305580 10207 raft_consensus.cc:2454] T 4885222f238c4d8ea20ebc4c7e171694 P e77d1c0d41034d1eb5a036dc17c071f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bdbecaeb8f214529b2012a9c003de2ff in term 0.
I20240502 01:59:31.305858 10021 leader_election.cc:302] T 4885222f238c4d8ea20ebc4c7e171694 P bdbecaeb8f214529b2012a9c003de2ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bdbecaeb8f214529b2012a9c003de2ff, e77d1c0d41034d1eb5a036dc17c071f5; no voters:
I20240502 01:59:31.306140 10263 raft_consensus.cc:2781] T 4885222f238c4d8ea20ebc4c7e171694 P bdbecaeb8f214529b2012a9c003de2ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:31.306210 10263 raft_consensus.cc:482] T 4885222f238c4d8ea20ebc4c7e171694 P bdbecaeb8f214529b2012a9c003de2ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:31.306281 10263 raft_consensus.cc:3037] T 4885222f238c4d8ea20ebc4c7e171694 P bdbecaeb8f214529b2012a9c003de2ff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:31.307137 10263 raft_consensus.cc:504] T 4885222f238c4d8ea20ebc4c7e171694 P bdbecaeb8f214529b2012a9c003de2ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.307507 10263 leader_election.cc:288] T 4885222f238c4d8ea20ebc4c7e171694 P bdbecaeb8f214529b2012a9c003de2ff [CANDIDATE]: Term 1 election: Requested vote from peers 4b7f7ba2ac084e829c4304c16a94894f (127.6.117.194:46805), e77d1c0d41034d1eb5a036dc17c071f5 (127.6.117.195:43977)
I20240502 01:59:31.307847 10207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4885222f238c4d8ea20ebc4c7e171694" candidate_uuid: "bdbecaeb8f214529b2012a9c003de2ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e77d1c0d41034d1eb5a036dc17c071f5"
I20240502 01:59:31.307880 10133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4885222f238c4d8ea20ebc4c7e171694" candidate_uuid: "bdbecaeb8f214529b2012a9c003de2ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b7f7ba2ac084e829c4304c16a94894f"
I20240502 01:59:31.307950 10207 raft_consensus.cc:3037] T 4885222f238c4d8ea20ebc4c7e171694 P e77d1c0d41034d1eb5a036dc17c071f5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:31.307972 10133 raft_consensus.cc:3037] T 4885222f238c4d8ea20ebc4c7e171694 P 4b7f7ba2ac084e829c4304c16a94894f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:31.308722 10207 raft_consensus.cc:2454] T 4885222f238c4d8ea20ebc4c7e171694 P e77d1c0d41034d1eb5a036dc17c071f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdbecaeb8f214529b2012a9c003de2ff in term 1.
I20240502 01:59:31.308750 10133 raft_consensus.cc:2454] T 4885222f238c4d8ea20ebc4c7e171694 P 4b7f7ba2ac084e829c4304c16a94894f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdbecaeb8f214529b2012a9c003de2ff in term 1.
I20240502 01:59:31.308987 10021 leader_election.cc:302] T 4885222f238c4d8ea20ebc4c7e171694 P bdbecaeb8f214529b2012a9c003de2ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bdbecaeb8f214529b2012a9c003de2ff, e77d1c0d41034d1eb5a036dc17c071f5; no voters:
I20240502 01:59:31.309219 10263 raft_consensus.cc:2781] T 4885222f238c4d8ea20ebc4c7e171694 P bdbecaeb8f214529b2012a9c003de2ff [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:31.309371 10263 raft_consensus.cc:686] T 4885222f238c4d8ea20ebc4c7e171694 P bdbecaeb8f214529b2012a9c003de2ff [term 1 LEADER]: Becoming Leader. State: Replica: bdbecaeb8f214529b2012a9c003de2ff, State: Running, Role: LEADER
I20240502 01:59:31.309530 10263 consensus_queue.cc:237] T 4885222f238c4d8ea20ebc4c7e171694 P bdbecaeb8f214529b2012a9c003de2ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.311529 9952 catalog_manager.cc:5484] T 4885222f238c4d8ea20ebc4c7e171694 P bdbecaeb8f214529b2012a9c003de2ff reported cstate change: term changed from 0 to 1, leader changed from <none> to bdbecaeb8f214529b2012a9c003de2ff (127.6.117.193). New cstate: current_term: 1 leader_uuid: "bdbecaeb8f214529b2012a9c003de2ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } health_report { overall_health: UNKNOWN } } }
I20240502 01:59:31.340471 10263 raft_consensus.cc:482] T 580db11a61cb46f48d5f7507cfbdbb52 P bdbecaeb8f214529b2012a9c003de2ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:31.340642 10263 raft_consensus.cc:504] T 580db11a61cb46f48d5f7507cfbdbb52 P bdbecaeb8f214529b2012a9c003de2ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.341094 10263 leader_election.cc:288] T 580db11a61cb46f48d5f7507cfbdbb52 P bdbecaeb8f214529b2012a9c003de2ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4b7f7ba2ac084e829c4304c16a94894f (127.6.117.194:46805), e77d1c0d41034d1eb5a036dc17c071f5 (127.6.117.195:43977)
I20240502 01:59:31.341585 10133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "580db11a61cb46f48d5f7507cfbdbb52" candidate_uuid: "bdbecaeb8f214529b2012a9c003de2ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b7f7ba2ac084e829c4304c16a94894f" is_pre_election: true
I20240502 01:59:31.341585 10207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "580db11a61cb46f48d5f7507cfbdbb52" candidate_uuid: "bdbecaeb8f214529b2012a9c003de2ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" is_pre_election: true
I20240502 01:59:31.341781 10207 raft_consensus.cc:2454] T 580db11a61cb46f48d5f7507cfbdbb52 P e77d1c0d41034d1eb5a036dc17c071f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bdbecaeb8f214529b2012a9c003de2ff in term 0.
I20240502 01:59:31.341799 10133 raft_consensus.cc:2454] T 580db11a61cb46f48d5f7507cfbdbb52 P 4b7f7ba2ac084e829c4304c16a94894f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bdbecaeb8f214529b2012a9c003de2ff in term 0.
I20240502 01:59:31.342088 10021 leader_election.cc:302] T 580db11a61cb46f48d5f7507cfbdbb52 P bdbecaeb8f214529b2012a9c003de2ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bdbecaeb8f214529b2012a9c003de2ff, e77d1c0d41034d1eb5a036dc17c071f5; no voters:
I20240502 01:59:31.342271 10263 raft_consensus.cc:2781] T 580db11a61cb46f48d5f7507cfbdbb52 P bdbecaeb8f214529b2012a9c003de2ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:31.342329 10263 raft_consensus.cc:482] T 580db11a61cb46f48d5f7507cfbdbb52 P bdbecaeb8f214529b2012a9c003de2ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:31.342355 10263 raft_consensus.cc:3037] T 580db11a61cb46f48d5f7507cfbdbb52 P bdbecaeb8f214529b2012a9c003de2ff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:31.343132 10263 raft_consensus.cc:504] T 580db11a61cb46f48d5f7507cfbdbb52 P bdbecaeb8f214529b2012a9c003de2ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.343472 10263 leader_election.cc:288] T 580db11a61cb46f48d5f7507cfbdbb52 P bdbecaeb8f214529b2012a9c003de2ff [CANDIDATE]: Term 1 election: Requested vote from peers 4b7f7ba2ac084e829c4304c16a94894f (127.6.117.194:46805), e77d1c0d41034d1eb5a036dc17c071f5 (127.6.117.195:43977)
I20240502 01:59:31.343837 10133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "580db11a61cb46f48d5f7507cfbdbb52" candidate_uuid: "bdbecaeb8f214529b2012a9c003de2ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b7f7ba2ac084e829c4304c16a94894f"
I20240502 01:59:31.343837 10207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "580db11a61cb46f48d5f7507cfbdbb52" candidate_uuid: "bdbecaeb8f214529b2012a9c003de2ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e77d1c0d41034d1eb5a036dc17c071f5"
I20240502 01:59:31.343974 10133 raft_consensus.cc:3037] T 580db11a61cb46f48d5f7507cfbdbb52 P 4b7f7ba2ac084e829c4304c16a94894f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:31.343974 10207 raft_consensus.cc:3037] T 580db11a61cb46f48d5f7507cfbdbb52 P e77d1c0d41034d1eb5a036dc17c071f5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:31.344913 10133 raft_consensus.cc:2454] T 580db11a61cb46f48d5f7507cfbdbb52 P 4b7f7ba2ac084e829c4304c16a94894f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdbecaeb8f214529b2012a9c003de2ff in term 1.
I20240502 01:59:31.344913 10207 raft_consensus.cc:2454] T 580db11a61cb46f48d5f7507cfbdbb52 P e77d1c0d41034d1eb5a036dc17c071f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdbecaeb8f214529b2012a9c003de2ff in term 1.
I20240502 01:59:31.345256 10022 leader_election.cc:302] T 580db11a61cb46f48d5f7507cfbdbb52 P bdbecaeb8f214529b2012a9c003de2ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b7f7ba2ac084e829c4304c16a94894f, bdbecaeb8f214529b2012a9c003de2ff; no voters:
I20240502 01:59:31.345433 10263 raft_consensus.cc:2781] T 580db11a61cb46f48d5f7507cfbdbb52 P bdbecaeb8f214529b2012a9c003de2ff [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:31.345526 10263 raft_consensus.cc:686] T 580db11a61cb46f48d5f7507cfbdbb52 P bdbecaeb8f214529b2012a9c003de2ff [term 1 LEADER]: Becoming Leader. State: Replica: bdbecaeb8f214529b2012a9c003de2ff, State: Running, Role: LEADER
I20240502 01:59:31.345654 10263 consensus_queue.cc:237] T 580db11a61cb46f48d5f7507cfbdbb52 P bdbecaeb8f214529b2012a9c003de2ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.347280 9952 catalog_manager.cc:5484] T 580db11a61cb46f48d5f7507cfbdbb52 P bdbecaeb8f214529b2012a9c003de2ff reported cstate change: term changed from 0 to 1, leader changed from <none> to bdbecaeb8f214529b2012a9c003de2ff (127.6.117.193). New cstate: current_term: 1 leader_uuid: "bdbecaeb8f214529b2012a9c003de2ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } health_report { overall_health: UNKNOWN } } }
I20240502 01:59:31.516345 10258 raft_consensus.cc:482] T 13d726dd71b14e84bc061f537f81e91d P 4b7f7ba2ac084e829c4304c16a94894f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:31.516507 10258 raft_consensus.cc:504] T 13d726dd71b14e84bc061f537f81e91d P 4b7f7ba2ac084e829c4304c16a94894f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.517005 10258 leader_election.cc:288] T 13d726dd71b14e84bc061f537f81e91d P 4b7f7ba2ac084e829c4304c16a94894f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bdbecaeb8f214529b2012a9c003de2ff (127.6.117.193:34597), e77d1c0d41034d1eb5a036dc17c071f5 (127.6.117.195:43977)
I20240502 01:59:31.517414 10207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13d726dd71b14e84bc061f537f81e91d" candidate_uuid: "4b7f7ba2ac084e829c4304c16a94894f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" is_pre_election: true
I20240502 01:59:31.517436 10057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13d726dd71b14e84bc061f537f81e91d" candidate_uuid: "4b7f7ba2ac084e829c4304c16a94894f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdbecaeb8f214529b2012a9c003de2ff" is_pre_election: true
I20240502 01:59:31.517545 10207 raft_consensus.cc:2454] T 13d726dd71b14e84bc061f537f81e91d P e77d1c0d41034d1eb5a036dc17c071f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b7f7ba2ac084e829c4304c16a94894f in term 0.
I20240502 01:59:31.517575 10057 raft_consensus.cc:2454] T 13d726dd71b14e84bc061f537f81e91d P bdbecaeb8f214529b2012a9c003de2ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b7f7ba2ac084e829c4304c16a94894f in term 0.
I20240502 01:59:31.517827 10097 leader_election.cc:302] T 13d726dd71b14e84bc061f537f81e91d P 4b7f7ba2ac084e829c4304c16a94894f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b7f7ba2ac084e829c4304c16a94894f, e77d1c0d41034d1eb5a036dc17c071f5; no voters:
I20240502 01:59:31.517998 10258 raft_consensus.cc:2781] T 13d726dd71b14e84bc061f537f81e91d P 4b7f7ba2ac084e829c4304c16a94894f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:31.518055 10258 raft_consensus.cc:482] T 13d726dd71b14e84bc061f537f81e91d P 4b7f7ba2ac084e829c4304c16a94894f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:31.518090 10258 raft_consensus.cc:3037] T 13d726dd71b14e84bc061f537f81e91d P 4b7f7ba2ac084e829c4304c16a94894f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:31.518878 10258 raft_consensus.cc:504] T 13d726dd71b14e84bc061f537f81e91d P 4b7f7ba2ac084e829c4304c16a94894f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.519186 10258 leader_election.cc:288] T 13d726dd71b14e84bc061f537f81e91d P 4b7f7ba2ac084e829c4304c16a94894f [CANDIDATE]: Term 1 election: Requested vote from peers bdbecaeb8f214529b2012a9c003de2ff (127.6.117.193:34597), e77d1c0d41034d1eb5a036dc17c071f5 (127.6.117.195:43977)
I20240502 01:59:31.519511 10207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13d726dd71b14e84bc061f537f81e91d" candidate_uuid: "4b7f7ba2ac084e829c4304c16a94894f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e77d1c0d41034d1eb5a036dc17c071f5"
I20240502 01:59:31.519511 10057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13d726dd71b14e84bc061f537f81e91d" candidate_uuid: "4b7f7ba2ac084e829c4304c16a94894f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdbecaeb8f214529b2012a9c003de2ff"
I20240502 01:59:31.519651 10207 raft_consensus.cc:3037] T 13d726dd71b14e84bc061f537f81e91d P e77d1c0d41034d1eb5a036dc17c071f5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:31.519668 10057 raft_consensus.cc:3037] T 13d726dd71b14e84bc061f537f81e91d P bdbecaeb8f214529b2012a9c003de2ff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:31.520538 10207 raft_consensus.cc:2454] T 13d726dd71b14e84bc061f537f81e91d P e77d1c0d41034d1eb5a036dc17c071f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b7f7ba2ac084e829c4304c16a94894f in term 1.
I20240502 01:59:31.520538 10057 raft_consensus.cc:2454] T 13d726dd71b14e84bc061f537f81e91d P bdbecaeb8f214529b2012a9c003de2ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b7f7ba2ac084e829c4304c16a94894f in term 1.
I20240502 01:59:31.520845 10099 leader_election.cc:302] T 13d726dd71b14e84bc061f537f81e91d P 4b7f7ba2ac084e829c4304c16a94894f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b7f7ba2ac084e829c4304c16a94894f, bdbecaeb8f214529b2012a9c003de2ff; no voters:
I20240502 01:59:31.520992 10258 raft_consensus.cc:2781] T 13d726dd71b14e84bc061f537f81e91d P 4b7f7ba2ac084e829c4304c16a94894f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:31.521075 10258 raft_consensus.cc:686] T 13d726dd71b14e84bc061f537f81e91d P 4b7f7ba2ac084e829c4304c16a94894f [term 1 LEADER]: Becoming Leader. State: Replica: 4b7f7ba2ac084e829c4304c16a94894f, State: Running, Role: LEADER
I20240502 01:59:31.521214 10258 consensus_queue.cc:237] T 13d726dd71b14e84bc061f537f81e91d P 4b7f7ba2ac084e829c4304c16a94894f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.522905 9952 catalog_manager.cc:5484] T 13d726dd71b14e84bc061f537f81e91d P 4b7f7ba2ac084e829c4304c16a94894f reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b7f7ba2ac084e829c4304c16a94894f (127.6.117.194). New cstate: current_term: 1 leader_uuid: "4b7f7ba2ac084e829c4304c16a94894f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } health_report { overall_health: UNKNOWN } } }
I20240502 01:59:31.565398 10057 raft_consensus.cc:1261] T b4409766ce994f698c31aede604a6927 P bdbecaeb8f214529b2012a9c003de2ff [term 1 FOLLOWER]: Refusing update from remote peer e77d1c0d41034d1eb5a036dc17c071f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:31.565615 10207 raft_consensus.cc:1261] T e7a8fc5afcab4b1db4ebf245b66df4be P e77d1c0d41034d1eb5a036dc17c071f5 [term 1 FOLLOWER]: Refusing update from remote peer bdbecaeb8f214529b2012a9c003de2ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:31.565865 10264 consensus_queue.cc:1035] T b4409766ce994f698c31aede604a6927 P e77d1c0d41034d1eb5a036dc17c071f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:31.569738 10132 raft_consensus.cc:1261] T b4409766ce994f698c31aede604a6927 P 4b7f7ba2ac084e829c4304c16a94894f [term 1 FOLLOWER]: Refusing update from remote peer e77d1c0d41034d1eb5a036dc17c071f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:31.573084 10056 raft_consensus.cc:1261] T 1851ea6467584b0ea32a1f68696803d0 P bdbecaeb8f214529b2012a9c003de2ff [term 1 FOLLOWER]: Refusing update from remote peer 4b7f7ba2ac084e829c4304c16a94894f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:31.573427 10055 raft_consensus.cc:1261] T 13d726dd71b14e84bc061f537f81e91d P bdbecaeb8f214529b2012a9c003de2ff [term 1 FOLLOWER]: Refusing update from remote peer 4b7f7ba2ac084e829c4304c16a94894f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:31.573630 10312 consensus_queue.cc:1035] T b4409766ce994f698c31aede604a6927 P e77d1c0d41034d1eb5a036dc17c071f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:31.576115 10208 raft_consensus.cc:1261] T abf7d8b411bb46b68ac9ae77d74f8e2b P e77d1c0d41034d1eb5a036dc17c071f5 [term 1 FOLLOWER]: Refusing update from remote peer bdbecaeb8f214529b2012a9c003de2ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:31.576442 10206 raft_consensus.cc:1261] T 4885222f238c4d8ea20ebc4c7e171694 P e77d1c0d41034d1eb5a036dc17c071f5 [term 1 FOLLOWER]: Refusing update from remote peer bdbecaeb8f214529b2012a9c003de2ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:31.576721 10205 raft_consensus.cc:1261] T 66ccad5a808b42378ccecc9da81cee74 P e77d1c0d41034d1eb5a036dc17c071f5 [term 1 FOLLOWER]: Refusing update from remote peer bdbecaeb8f214529b2012a9c003de2ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:31.577003 10204 raft_consensus.cc:1261] T 580db11a61cb46f48d5f7507cfbdbb52 P e77d1c0d41034d1eb5a036dc17c071f5 [term 1 FOLLOWER]: Refusing update from remote peer bdbecaeb8f214529b2012a9c003de2ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:31.577275 10203 raft_consensus.cc:1261] T bceb844e50b84e278c0c49f4f4912e00 P e77d1c0d41034d1eb5a036dc17c071f5 [term 1 FOLLOWER]: Refusing update from remote peer 4b7f7ba2ac084e829c4304c16a94894f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:31.565923 10133 raft_consensus.cc:1261] T e7a8fc5afcab4b1db4ebf245b66df4be P 4b7f7ba2ac084e829c4304c16a94894f [term 1 FOLLOWER]: Refusing update from remote peer bdbecaeb8f214529b2012a9c003de2ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:31.578629 10057 raft_consensus.cc:1261] T bceb844e50b84e278c0c49f4f4912e00 P bdbecaeb8f214529b2012a9c003de2ff [term 1 FOLLOWER]: Refusing update from remote peer 4b7f7ba2ac084e829c4304c16a94894f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:31.579133 10307 consensus_queue.cc:1035] T 13d726dd71b14e84bc061f537f81e91d P 4b7f7ba2ac084e829c4304c16a94894f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:31.579263 10055 raft_consensus.cc:1261] T 73ea0bfb40ad43a68a6289ff1b35d005 P bdbecaeb8f214529b2012a9c003de2ff [term 1 FOLLOWER]: Refusing update from remote peer 4b7f7ba2ac084e829c4304c16a94894f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:31.579527 10207 raft_consensus.cc:1261] T 39f1842ff1494d78bfa0811a799c4bb8 P e77d1c0d41034d1eb5a036dc17c071f5 [term 1 FOLLOWER]: Refusing update from remote peer bdbecaeb8f214529b2012a9c003de2ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:31.579114 10304 consensus_queue.cc:1035] T 1851ea6467584b0ea32a1f68696803d0 P 4b7f7ba2ac084e829c4304c16a94894f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 01:59:31.580540 10056 raft_consensus.cc:1261] T 19ff9c216a094fde86ea9f9648bf08cb P bdbecaeb8f214529b2012a9c003de2ff [term 1 FOLLOWER]: Refusing update from remote peer 4b7f7ba2ac084e829c4304c16a94894f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:31.580817 10298 consensus_queue.cc:1035] T e7a8fc5afcab4b1db4ebf245b66df4be P bdbecaeb8f214529b2012a9c003de2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:31.581506 10304 consensus_queue.cc:1035] T bceb844e50b84e278c0c49f4f4912e00 P 4b7f7ba2ac084e829c4304c16a94894f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:31.581590 10258 consensus_queue.cc:1035] T 73ea0bfb40ad43a68a6289ff1b35d005 P 4b7f7ba2ac084e829c4304c16a94894f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:31.581795 10304 consensus_queue.cc:1035] T 19ff9c216a094fde86ea9f9648bf08cb P 4b7f7ba2ac084e829c4304c16a94894f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:31.582221 10205 raft_consensus.cc:1261] T 73ea0bfb40ad43a68a6289ff1b35d005 P e77d1c0d41034d1eb5a036dc17c071f5 [term 1 FOLLOWER]: Refusing update from remote peer 4b7f7ba2ac084e829c4304c16a94894f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:31.582309 10204 raft_consensus.cc:1261] T 13d726dd71b14e84bc061f537f81e91d P e77d1c0d41034d1eb5a036dc17c071f5 [term 1 FOLLOWER]: Refusing update from remote peer 4b7f7ba2ac084e829c4304c16a94894f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:31.582485 10206 raft_consensus.cc:1261] T 19ff9c216a094fde86ea9f9648bf08cb P e77d1c0d41034d1eb5a036dc17c071f5 [term 1 FOLLOWER]: Refusing update from remote peer 4b7f7ba2ac084e829c4304c16a94894f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:31.582581 10203 raft_consensus.cc:1261] T 1851ea6467584b0ea32a1f68696803d0 P e77d1c0d41034d1eb5a036dc17c071f5 [term 1 FOLLOWER]: Refusing update from remote peer 4b7f7ba2ac084e829c4304c16a94894f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:31.583216 10258 consensus_queue.cc:1035] T bceb844e50b84e278c0c49f4f4912e00 P 4b7f7ba2ac084e829c4304c16a94894f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:31.585155 10258 consensus_queue.cc:1035] T 73ea0bfb40ad43a68a6289ff1b35d005 P 4b7f7ba2ac084e829c4304c16a94894f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:31.585356 10307 consensus_queue.cc:1035] T 13d726dd71b14e84bc061f537f81e91d P 4b7f7ba2ac084e829c4304c16a94894f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:31.585425 10258 consensus_queue.cc:1035] T 19ff9c216a094fde86ea9f9648bf08cb P 4b7f7ba2ac084e829c4304c16a94894f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:31.585592 10307 consensus_queue.cc:1035] T 1851ea6467584b0ea32a1f68696803d0 P 4b7f7ba2ac084e829c4304c16a94894f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:31.585902 10356 consensus_queue.cc:1035] T abf7d8b411bb46b68ac9ae77d74f8e2b P bdbecaeb8f214529b2012a9c003de2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:31.586066 10298 consensus_queue.cc:1035] T 4885222f238c4d8ea20ebc4c7e171694 P bdbecaeb8f214529b2012a9c003de2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:31.586369 10298 consensus_queue.cc:1035] T 66ccad5a808b42378ccecc9da81cee74 P bdbecaeb8f214529b2012a9c003de2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:31.586421 10356 consensus_queue.cc:1035] T 580db11a61cb46f48d5f7507cfbdbb52 P bdbecaeb8f214529b2012a9c003de2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:31.586637 10298 consensus_queue.cc:1035] T 39f1842ff1494d78bfa0811a799c4bb8 P bdbecaeb8f214529b2012a9c003de2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:31.587548 10131 raft_consensus.cc:1261] T abf7d8b411bb46b68ac9ae77d74f8e2b P 4b7f7ba2ac084e829c4304c16a94894f [term 1 FOLLOWER]: Refusing update from remote peer bdbecaeb8f214529b2012a9c003de2ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:31.587826 10130 raft_consensus.cc:1261] T 4885222f238c4d8ea20ebc4c7e171694 P 4b7f7ba2ac084e829c4304c16a94894f [term 1 FOLLOWER]: Refusing update from remote peer bdbecaeb8f214529b2012a9c003de2ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:31.588111 10129 raft_consensus.cc:1261] T 66ccad5a808b42378ccecc9da81cee74 P 4b7f7ba2ac084e829c4304c16a94894f [term 1 FOLLOWER]: Refusing update from remote peer bdbecaeb8f214529b2012a9c003de2ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:31.588397 10128 raft_consensus.cc:1261] T 580db11a61cb46f48d5f7507cfbdbb52 P 4b7f7ba2ac084e829c4304c16a94894f [term 1 FOLLOWER]: Refusing update from remote peer bdbecaeb8f214529b2012a9c003de2ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:31.588632 10133 raft_consensus.cc:1261] T 39f1842ff1494d78bfa0811a799c4bb8 P 4b7f7ba2ac084e829c4304c16a94894f [term 1 FOLLOWER]: Refusing update from remote peer bdbecaeb8f214529b2012a9c003de2ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:31.597148 10298 consensus_queue.cc:1035] T e7a8fc5afcab4b1db4ebf245b66df4be P bdbecaeb8f214529b2012a9c003de2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:31.597530 10298 consensus_queue.cc:1035] T abf7d8b411bb46b68ac9ae77d74f8e2b P bdbecaeb8f214529b2012a9c003de2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:31.597836 10298 consensus_queue.cc:1035] T 4885222f238c4d8ea20ebc4c7e171694 P bdbecaeb8f214529b2012a9c003de2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:31.598109 10298 consensus_queue.cc:1035] T 66ccad5a808b42378ccecc9da81cee74 P bdbecaeb8f214529b2012a9c003de2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:31.598374 10298 consensus_queue.cc:1035] T 580db11a61cb46f48d5f7507cfbdbb52 P bdbecaeb8f214529b2012a9c003de2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:31.598629 10298 consensus_queue.cc:1035] T 39f1842ff1494d78bfa0811a799c4bb8 P bdbecaeb8f214529b2012a9c003de2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:31.602624 10368 mvcc.cc:205] Tried to move back new op lower bound from 7023063742782808064 to 7023063741348282368. Current Snapshot: MvccSnapshot[applied={T|T < 7023063742759768064}]
I20240502 01:59:31.620666 10328 mvcc.cc:205] Tried to move back new op lower bound from 7023063742764564480 to 7023063741685657600. Current Snapshot: MvccSnapshot[applied={T|T < 7023063742735093760}]
I20240502 01:59:31.672981 10331 mvcc.cc:205] Tried to move back new op lower bound from 7023063742793207808 to 7023063741467426816. Current Snapshot: MvccSnapshot[applied={T|T < 7023063742753955840}]
I20240502 01:59:31.895363 10008 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:59:31.940011 10130 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "13d726dd71b14e84bc061f537f81e91d"
dest_uuid: "4b7f7ba2ac084e829c4304c16a94894f"
mode: GRACEFUL
new_leader_uuid: "e77d1c0d41034d1eb5a036dc17c071f5"
from {username='slave'} at 127.0.0.1:52630
I20240502 01:59:31.940183 10130 raft_consensus.cc:595] T 13d726dd71b14e84bc061f537f81e91d P 4b7f7ba2ac084e829c4304c16a94894f [term 1 LEADER]: Received request to transfer leadership to TS e77d1c0d41034d1eb5a036dc17c071f5
W20240502 01:59:31.951368 10107 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52624: Service unavailable: leader transfer in progress
I20240502 01:59:31.951826 10053 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4885222f238c4d8ea20ebc4c7e171694"
dest_uuid: "bdbecaeb8f214529b2012a9c003de2ff"
mode: GRACEFUL
new_leader_uuid: "e77d1c0d41034d1eb5a036dc17c071f5"
from {username='slave'} at 127.0.0.1:44420
I20240502 01:59:31.951972 10053 raft_consensus.cc:595] T 4885222f238c4d8ea20ebc4c7e171694 P bdbecaeb8f214529b2012a9c003de2ff [term 1 LEADER]: Received request to transfer leadership to TS e77d1c0d41034d1eb5a036dc17c071f5
W20240502 01:59:31.952100 10032 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44412: Service unavailable: leader transfer in progress
I20240502 01:59:31.952760 10053 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "66ccad5a808b42378ccecc9da81cee74"
dest_uuid: "bdbecaeb8f214529b2012a9c003de2ff"
mode: GRACEFUL
new_leader_uuid: "e77d1c0d41034d1eb5a036dc17c071f5"
from {username='slave'} at 127.0.0.1:44420
I20240502 01:59:31.952870 10053 raft_consensus.cc:595] T 66ccad5a808b42378ccecc9da81cee74 P bdbecaeb8f214529b2012a9c003de2ff [term 1 LEADER]: Received request to transfer leadership to TS e77d1c0d41034d1eb5a036dc17c071f5
I20240502 01:59:31.953325 10008 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 3
I20240502 01:59:31.953461 10008 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
W20240502 01:59:31.953958 10032 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44412: Service unavailable: leader transfer in progress
W20240502 01:59:31.954851 10107 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52624: Service unavailable: leader transfer in progress
I20240502 01:59:31.957938 10390 raft_consensus.cc:979] T 66ccad5a808b42378ccecc9da81cee74 P bdbecaeb8f214529b2012a9c003de2ff: : Instructing follower e77d1c0d41034d1eb5a036dc17c071f5 to start an election
I20240502 01:59:31.958036 10390 raft_consensus.cc:1067] T 66ccad5a808b42378ccecc9da81cee74 P bdbecaeb8f214529b2012a9c003de2ff [term 1 LEADER]: Signalling peer e77d1c0d41034d1eb5a036dc17c071f5 to start an election
I20240502 01:59:31.958294 10393 raft_consensus.cc:979] T 13d726dd71b14e84bc061f537f81e91d P 4b7f7ba2ac084e829c4304c16a94894f: : Instructing follower e77d1c0d41034d1eb5a036dc17c071f5 to start an election
I20240502 01:59:31.958379 10393 raft_consensus.cc:1067] T 13d726dd71b14e84bc061f537f81e91d P 4b7f7ba2ac084e829c4304c16a94894f [term 1 LEADER]: Signalling peer e77d1c0d41034d1eb5a036dc17c071f5 to start an election
I20240502 01:59:31.959084 10355 raft_consensus.cc:979] T 4885222f238c4d8ea20ebc4c7e171694 P bdbecaeb8f214529b2012a9c003de2ff: : Instructing follower e77d1c0d41034d1eb5a036dc17c071f5 to start an election
I20240502 01:59:31.959163 10355 raft_consensus.cc:1067] T 4885222f238c4d8ea20ebc4c7e171694 P bdbecaeb8f214529b2012a9c003de2ff [term 1 LEADER]: Signalling peer e77d1c0d41034d1eb5a036dc17c071f5 to start an election
I20240502 01:59:31.960604 10202 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "13d726dd71b14e84bc061f537f81e91d"
dest_uuid: "e77d1c0d41034d1eb5a036dc17c071f5"
from {username='slave'} at 127.0.0.1:48076
I20240502 01:59:31.960868 10202 raft_consensus.cc:482] T 13d726dd71b14e84bc061f537f81e91d P e77d1c0d41034d1eb5a036dc17c071f5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:59:31.960999 10202 raft_consensus.cc:3037] T 13d726dd71b14e84bc061f537f81e91d P e77d1c0d41034d1eb5a036dc17c071f5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:59:31.962249 10202 raft_consensus.cc:504] T 13d726dd71b14e84bc061f537f81e91d P e77d1c0d41034d1eb5a036dc17c071f5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.962872 10054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13d726dd71b14e84bc061f537f81e91d" candidate_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 20 } } ignore_live_leader: true dest_uuid: "bdbecaeb8f214529b2012a9c003de2ff"
I20240502 01:59:31.963039 10054 raft_consensus.cc:3037] T 13d726dd71b14e84bc061f537f81e91d P bdbecaeb8f214529b2012a9c003de2ff [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:59:31.964033 10054 raft_consensus.cc:2454] T 13d726dd71b14e84bc061f537f81e91d P bdbecaeb8f214529b2012a9c003de2ff [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e77d1c0d41034d1eb5a036dc17c071f5 in term 2.
I20240502 01:59:31.964318 10174 leader_election.cc:302] T 13d726dd71b14e84bc061f537f81e91d P e77d1c0d41034d1eb5a036dc17c071f5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bdbecaeb8f214529b2012a9c003de2ff, e77d1c0d41034d1eb5a036dc17c071f5; no voters:
I20240502 01:59:31.964491 10383 raft_consensus.cc:2781] T 13d726dd71b14e84bc061f537f81e91d P e77d1c0d41034d1eb5a036dc17c071f5 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:59:31.964563 10383 raft_consensus.cc:686] T 13d726dd71b14e84bc061f537f81e91d P e77d1c0d41034d1eb5a036dc17c071f5 [term 2 LEADER]: Becoming Leader. State: Replica: e77d1c0d41034d1eb5a036dc17c071f5, State: Running, Role: LEADER
I20240502 01:59:31.964733 10383 consensus_queue.cc:237] T 13d726dd71b14e84bc061f537f81e91d P e77d1c0d41034d1eb5a036dc17c071f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 1.20, Last appended by leader: 20, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.966535 9947 catalog_manager.cc:5484] T 13d726dd71b14e84bc061f537f81e91d P e77d1c0d41034d1eb5a036dc17c071f5 reported cstate change: term changed from 1 to 2, leader changed from 4b7f7ba2ac084e829c4304c16a94894f (127.6.117.194) to e77d1c0d41034d1eb5a036dc17c071f5 (127.6.117.195). New cstate: current_term: 2 leader_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } health_report { overall_health: HEALTHY } } }
I20240502 01:59:31.967052 10208 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "66ccad5a808b42378ccecc9da81cee74"
dest_uuid: "e77d1c0d41034d1eb5a036dc17c071f5"
from {username='slave'} at 127.0.0.1:48064
I20240502 01:59:31.967157 10208 raft_consensus.cc:482] T 66ccad5a808b42378ccecc9da81cee74 P e77d1c0d41034d1eb5a036dc17c071f5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:59:31.967211 10208 raft_consensus.cc:3037] T 66ccad5a808b42378ccecc9da81cee74 P e77d1c0d41034d1eb5a036dc17c071f5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:59:31.968232 10208 raft_consensus.cc:504] T 66ccad5a808b42378ccecc9da81cee74 P e77d1c0d41034d1eb5a036dc17c071f5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.968626 10208 leader_election.cc:288] T 66ccad5a808b42378ccecc9da81cee74 P e77d1c0d41034d1eb5a036dc17c071f5 [CANDIDATE]: Term 2 election: Requested vote from peers bdbecaeb8f214529b2012a9c003de2ff (127.6.117.193:34597), 4b7f7ba2ac084e829c4304c16a94894f (127.6.117.194:46805)
I20240502 01:59:31.968928 10129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66ccad5a808b42378ccecc9da81cee74" candidate_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 22 } } ignore_live_leader: true dest_uuid: "4b7f7ba2ac084e829c4304c16a94894f"
I20240502 01:59:31.969051 10129 raft_consensus.cc:3037] T 66ccad5a808b42378ccecc9da81cee74 P 4b7f7ba2ac084e829c4304c16a94894f [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:59:31.969230 10056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66ccad5a808b42378ccecc9da81cee74" candidate_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 22 } } ignore_live_leader: true dest_uuid: "bdbecaeb8f214529b2012a9c003de2ff"
I20240502 01:59:31.969333 10056 raft_consensus.cc:3032] T 66ccad5a808b42378ccecc9da81cee74 P bdbecaeb8f214529b2012a9c003de2ff [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:59:31.969372 10056 raft_consensus.cc:729] T 66ccad5a808b42378ccecc9da81cee74 P bdbecaeb8f214529b2012a9c003de2ff [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bdbecaeb8f214529b2012a9c003de2ff, State: Running, Role: LEADER
I20240502 01:59:31.969509 10056 consensus_queue.cc:260] T 66ccad5a808b42378ccecc9da81cee74 P bdbecaeb8f214529b2012a9c003de2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 1.22, Last appended by leader: 22, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.969621 10056 raft_consensus.cc:3037] T 66ccad5a808b42378ccecc9da81cee74 P bdbecaeb8f214529b2012a9c003de2ff [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:59:31.970082 10129 raft_consensus.cc:2454] T 66ccad5a808b42378ccecc9da81cee74 P 4b7f7ba2ac084e829c4304c16a94894f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e77d1c0d41034d1eb5a036dc17c071f5 in term 2.
I20240502 01:59:31.970336 10173 leader_election.cc:302] T 66ccad5a808b42378ccecc9da81cee74 P e77d1c0d41034d1eb5a036dc17c071f5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b7f7ba2ac084e829c4304c16a94894f, e77d1c0d41034d1eb5a036dc17c071f5; no voters:
I20240502 01:59:31.970533 10056 raft_consensus.cc:2454] T 66ccad5a808b42378ccecc9da81cee74 P bdbecaeb8f214529b2012a9c003de2ff [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e77d1c0d41034d1eb5a036dc17c071f5 in term 2.
I20240502 01:59:31.970857 10382 raft_consensus.cc:2781] T 66ccad5a808b42378ccecc9da81cee74 P e77d1c0d41034d1eb5a036dc17c071f5 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:59:31.970942 10382 raft_consensus.cc:686] T 66ccad5a808b42378ccecc9da81cee74 P e77d1c0d41034d1eb5a036dc17c071f5 [term 2 LEADER]: Becoming Leader. State: Replica: e77d1c0d41034d1eb5a036dc17c071f5, State: Running, Role: LEADER
I20240502 01:59:31.971073 10382 consensus_queue.cc:237] T 66ccad5a808b42378ccecc9da81cee74 P e77d1c0d41034d1eb5a036dc17c071f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 1.22, Last appended by leader: 22, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.972826 9947 catalog_manager.cc:5484] T 66ccad5a808b42378ccecc9da81cee74 P e77d1c0d41034d1eb5a036dc17c071f5 reported cstate change: term changed from 1 to 2, leader changed from bdbecaeb8f214529b2012a9c003de2ff (127.6.117.193) to e77d1c0d41034d1eb5a036dc17c071f5 (127.6.117.195). New cstate: current_term: 2 leader_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } health_report { overall_health: HEALTHY } } }
I20240502 01:59:31.974998 10202 leader_election.cc:288] T 13d726dd71b14e84bc061f537f81e91d P e77d1c0d41034d1eb5a036dc17c071f5 [CANDIDATE]: Term 2 election: Requested vote from peers bdbecaeb8f214529b2012a9c003de2ff (127.6.117.193:34597), 4b7f7ba2ac084e829c4304c16a94894f (127.6.117.194:46805)
I20240502 01:59:31.975256 10129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13d726dd71b14e84bc061f537f81e91d" candidate_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 20 } } ignore_live_leader: true dest_uuid: "4b7f7ba2ac084e829c4304c16a94894f"
I20240502 01:59:31.975358 10129 raft_consensus.cc:3032] T 13d726dd71b14e84bc061f537f81e91d P 4b7f7ba2ac084e829c4304c16a94894f [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:59:31.975394 10129 raft_consensus.cc:729] T 13d726dd71b14e84bc061f537f81e91d P 4b7f7ba2ac084e829c4304c16a94894f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4b7f7ba2ac084e829c4304c16a94894f, State: Running, Role: LEADER
I20240502 01:59:31.975530 10129 consensus_queue.cc:260] T 13d726dd71b14e84bc061f537f81e91d P 4b7f7ba2ac084e829c4304c16a94894f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 1.20, Last appended by leader: 20, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.975642 10129 raft_consensus.cc:3037] T 13d726dd71b14e84bc061f537f81e91d P 4b7f7ba2ac084e829c4304c16a94894f [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:59:31.975795 10202 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4885222f238c4d8ea20ebc4c7e171694"
dest_uuid: "e77d1c0d41034d1eb5a036dc17c071f5"
from {username='slave'} at 127.0.0.1:48064
I20240502 01:59:31.975901 10202 raft_consensus.cc:482] T 4885222f238c4d8ea20ebc4c7e171694 P e77d1c0d41034d1eb5a036dc17c071f5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:59:31.975960 10202 raft_consensus.cc:3037] T 4885222f238c4d8ea20ebc4c7e171694 P e77d1c0d41034d1eb5a036dc17c071f5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:59:31.976621 10129 raft_consensus.cc:2454] T 13d726dd71b14e84bc061f537f81e91d P 4b7f7ba2ac084e829c4304c16a94894f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e77d1c0d41034d1eb5a036dc17c071f5 in term 2.
I20240502 01:59:31.976946 10202 raft_consensus.cc:504] T 4885222f238c4d8ea20ebc4c7e171694 P e77d1c0d41034d1eb5a036dc17c071f5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.977855 10202 leader_election.cc:288] T 4885222f238c4d8ea20ebc4c7e171694 P e77d1c0d41034d1eb5a036dc17c071f5 [CANDIDATE]: Term 2 election: Requested vote from peers 4b7f7ba2ac084e829c4304c16a94894f (127.6.117.194:46805), bdbecaeb8f214529b2012a9c003de2ff (127.6.117.193:34597)
I20240502 01:59:31.978119 10053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4885222f238c4d8ea20ebc4c7e171694" candidate_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 22 } } ignore_live_leader: true dest_uuid: "bdbecaeb8f214529b2012a9c003de2ff"
I20240502 01:59:31.978224 10053 raft_consensus.cc:3032] T 4885222f238c4d8ea20ebc4c7e171694 P bdbecaeb8f214529b2012a9c003de2ff [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:59:31.978263 10053 raft_consensus.cc:729] T 4885222f238c4d8ea20ebc4c7e171694 P bdbecaeb8f214529b2012a9c003de2ff [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bdbecaeb8f214529b2012a9c003de2ff, State: Running, Role: LEADER
I20240502 01:59:31.978394 10053 consensus_queue.cc:260] T 4885222f238c4d8ea20ebc4c7e171694 P bdbecaeb8f214529b2012a9c003de2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 1.22, Last appended by leader: 22, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.978503 10053 raft_consensus.cc:3037] T 4885222f238c4d8ea20ebc4c7e171694 P bdbecaeb8f214529b2012a9c003de2ff [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:59:31.980072 10053 raft_consensus.cc:2454] T 4885222f238c4d8ea20ebc4c7e171694 P bdbecaeb8f214529b2012a9c003de2ff [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e77d1c0d41034d1eb5a036dc17c071f5 in term 2.
I20240502 01:59:31.980399 10174 leader_election.cc:302] T 4885222f238c4d8ea20ebc4c7e171694 P e77d1c0d41034d1eb5a036dc17c071f5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bdbecaeb8f214529b2012a9c003de2ff, e77d1c0d41034d1eb5a036dc17c071f5; no voters:
I20240502 01:59:31.980584 10369 raft_consensus.cc:2781] T 4885222f238c4d8ea20ebc4c7e171694 P e77d1c0d41034d1eb5a036dc17c071f5 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:59:31.980664 10369 raft_consensus.cc:686] T 4885222f238c4d8ea20ebc4c7e171694 P e77d1c0d41034d1eb5a036dc17c071f5 [term 2 LEADER]: Becoming Leader. State: Replica: e77d1c0d41034d1eb5a036dc17c071f5, State: Running, Role: LEADER
I20240502 01:59:31.980774 10369 consensus_queue.cc:237] T 4885222f238c4d8ea20ebc4c7e171694 P e77d1c0d41034d1eb5a036dc17c071f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 1.22, Last appended by leader: 22, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:31.981328 10054 raft_consensus.cc:1226] T 13d726dd71b14e84bc061f537f81e91d P bdbecaeb8f214529b2012a9c003de2ff [term 2 FOLLOWER]: Rejecting Update request from peer 4b7f7ba2ac084e829c4304c16a94894f for earlier term 1. Current term is 2. Ops: [1.20-1.20]
I20240502 01:59:31.981631 10129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4885222f238c4d8ea20ebc4c7e171694" candidate_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 22 } } ignore_live_leader: true dest_uuid: "4b7f7ba2ac084e829c4304c16a94894f"
I20240502 01:59:31.981719 10129 raft_consensus.cc:3037] T 4885222f238c4d8ea20ebc4c7e171694 P 4b7f7ba2ac084e829c4304c16a94894f [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:59:31.982340 9947 catalog_manager.cc:5484] T 4885222f238c4d8ea20ebc4c7e171694 P e77d1c0d41034d1eb5a036dc17c071f5 reported cstate change: term changed from 1 to 2, leader changed from bdbecaeb8f214529b2012a9c003de2ff (127.6.117.193) to e77d1c0d41034d1eb5a036dc17c071f5 (127.6.117.195). New cstate: current_term: 2 leader_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } health_report { overall_health: HEALTHY } } }
I20240502 01:59:31.982436 10129 raft_consensus.cc:2454] T 4885222f238c4d8ea20ebc4c7e171694 P 4b7f7ba2ac084e829c4304c16a94894f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e77d1c0d41034d1eb5a036dc17c071f5 in term 2.
I20240502 01:59:31.986423 10131 raft_consensus.cc:1226] T 66ccad5a808b42378ccecc9da81cee74 P 4b7f7ba2ac084e829c4304c16a94894f [term 2 FOLLOWER]: Rejecting Update request from peer bdbecaeb8f214529b2012a9c003de2ff for earlier term 1. Current term is 2. Ops: [1.20-1.20]
I20240502 01:59:31.987231 10131 raft_consensus.cc:1226] T 4885222f238c4d8ea20ebc4c7e171694 P 4b7f7ba2ac084e829c4304c16a94894f [term 2 FOLLOWER]: Rejecting Update request from peer bdbecaeb8f214529b2012a9c003de2ff for earlier term 1. Current term is 2. Ops: [1.20-1.20]
W20240502 01:59:31.990825 10033 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44412: Illegal state: replica bdbecaeb8f214529b2012a9c003de2ff is not leader of this config: current role FOLLOWER
W20240502 01:59:31.990968 10032 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44412: Illegal state: replica bdbecaeb8f214529b2012a9c003de2ff is not leader of this config: current role FOLLOWER
W20240502 01:59:31.992615 10105 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52624: Illegal state: replica 4b7f7ba2ac084e829c4304c16a94894f is not leader of this config: current role FOLLOWER
W20240502 01:59:31.992642 10029 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44412: Illegal state: replica bdbecaeb8f214529b2012a9c003de2ff is not leader of this config: current role FOLLOWER
W20240502 01:59:31.993155 10029 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44412: Illegal state: replica bdbecaeb8f214529b2012a9c003de2ff is not leader of this config: current role FOLLOWER
W20240502 01:59:31.993974 10105 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52624: Illegal state: replica 4b7f7ba2ac084e829c4304c16a94894f is not leader of this config: current role FOLLOWER
W20240502 01:59:31.993973 10107 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52624: Illegal state: replica 4b7f7ba2ac084e829c4304c16a94894f is not leader of this config: current role FOLLOWER
W20240502 01:59:31.997023 10107 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52624: Illegal state: replica 4b7f7ba2ac084e829c4304c16a94894f is not leader of this config: current role FOLLOWER
W20240502 01:59:31.998656 10033 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44412: Illegal state: replica bdbecaeb8f214529b2012a9c003de2ff is not leader of this config: current role FOLLOWER
W20240502 01:59:32.003122 10033 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44412: Illegal state: replica bdbecaeb8f214529b2012a9c003de2ff is not leader of this config: current role FOLLOWER
W20240502 01:59:32.004976 10029 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44412: Illegal state: replica bdbecaeb8f214529b2012a9c003de2ff is not leader of this config: current role FOLLOWER
W20240502 01:59:32.005601 10029 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44412: Illegal state: replica bdbecaeb8f214529b2012a9c003de2ff is not leader of this config: current role FOLLOWER
W20240502 01:59:32.014961 10104 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52624: Illegal state: replica 4b7f7ba2ac084e829c4304c16a94894f is not leader of this config: current role FOLLOWER
W20240502 01:59:32.014950 10107 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52624: Illegal state: replica 4b7f7ba2ac084e829c4304c16a94894f is not leader of this config: current role FOLLOWER
W20240502 01:59:32.015746 10107 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52624: Illegal state: replica 4b7f7ba2ac084e829c4304c16a94894f is not leader of this config: current role FOLLOWER
I20240502 01:59:32.015954 10053 raft_consensus.cc:1261] T 13d726dd71b14e84bc061f537f81e91d P bdbecaeb8f214529b2012a9c003de2ff [term 2 FOLLOWER]: Refusing update from remote peer e77d1c0d41034d1eb5a036dc17c071f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 19. Preceding OpId from leader: term: 2 index: 22. (index mismatch)
W20240502 01:59:32.016206 10107 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52624: Illegal state: replica 4b7f7ba2ac084e829c4304c16a94894f is not leader of this config: current role FOLLOWER
I20240502 01:59:32.016427 10383 consensus_queue.cc:1035] T 13d726dd71b14e84bc061f537f81e91d P e77d1c0d41034d1eb5a036dc17c071f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.000s
W20240502 01:59:32.016758 10107 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52624: Illegal state: replica 4b7f7ba2ac084e829c4304c16a94894f is not leader of this config: current role FOLLOWER
I20240502 01:59:32.018345 10129 raft_consensus.cc:1261] T 13d726dd71b14e84bc061f537f81e91d P 4b7f7ba2ac084e829c4304c16a94894f [term 2 FOLLOWER]: Refusing update from remote peer e77d1c0d41034d1eb5a036dc17c071f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 20. Preceding OpId from leader: term: 2 index: 22. (index mismatch)
I20240502 01:59:32.018713 10383 consensus_queue.cc:1035] T 13d726dd71b14e84bc061f537f81e91d P e77d1c0d41034d1eb5a036dc17c071f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
W20240502 01:59:32.029080 10029 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44412: Illegal state: replica bdbecaeb8f214529b2012a9c003de2ff is not leader of this config: current role FOLLOWER
W20240502 01:59:32.029687 10029 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44412: Illegal state: replica bdbecaeb8f214529b2012a9c003de2ff is not leader of this config: current role FOLLOWER
W20240502 01:59:32.030727 10107 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52624: Illegal state: replica 4b7f7ba2ac084e829c4304c16a94894f is not leader of this config: current role FOLLOWER
I20240502 01:59:32.030938 10055 raft_consensus.cc:1261] T 4885222f238c4d8ea20ebc4c7e171694 P bdbecaeb8f214529b2012a9c003de2ff [term 2 FOLLOWER]: Refusing update from remote peer e77d1c0d41034d1eb5a036dc17c071f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 22. Preceding OpId from leader: term: 2 index: 24. (index mismatch)
I20240502 01:59:32.031394 10382 consensus_queue.cc:1035] T 4885222f238c4d8ea20ebc4c7e171694 P e77d1c0d41034d1eb5a036dc17c071f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20240502 01:59:32.033941 10055 raft_consensus.cc:1261] T 66ccad5a808b42378ccecc9da81cee74 P bdbecaeb8f214529b2012a9c003de2ff [term 2 FOLLOWER]: Refusing update from remote peer e77d1c0d41034d1eb5a036dc17c071f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 22. Preceding OpId from leader: term: 2 index: 24. (index mismatch)
I20240502 01:59:32.036765 10264 consensus_queue.cc:1035] T 66ccad5a808b42378ccecc9da81cee74 P e77d1c0d41034d1eb5a036dc17c071f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20240502 01:59:32.042778 10133 raft_consensus.cc:1261] T 4885222f238c4d8ea20ebc4c7e171694 P 4b7f7ba2ac084e829c4304c16a94894f [term 2 FOLLOWER]: Refusing update from remote peer e77d1c0d41034d1eb5a036dc17c071f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 19. Preceding OpId from leader: term: 2 index: 24. (index mismatch)
I20240502 01:59:32.043576 10129 raft_consensus.cc:1261] T 66ccad5a808b42378ccecc9da81cee74 P 4b7f7ba2ac084e829c4304c16a94894f [term 2 FOLLOWER]: Refusing update from remote peer e77d1c0d41034d1eb5a036dc17c071f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 19. Preceding OpId from leader: term: 2 index: 24. (index mismatch)
I20240502 01:59:32.043788 10312 consensus_queue.cc:1035] T 4885222f238c4d8ea20ebc4c7e171694 P e77d1c0d41034d1eb5a036dc17c071f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 19, Time since last communication: 0.000s
I20240502 01:59:32.044065 10264 consensus_queue.cc:1035] T 66ccad5a808b42378ccecc9da81cee74 P e77d1c0d41034d1eb5a036dc17c071f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 19, Time since last communication: 0.000s
I20240502 01:59:32.046447 10334 mvcc.cc:205] Tried to move back new op lower bound from 7023063744661676032 to 7023063744394952704. Current Snapshot: MvccSnapshot[applied={T|T < 7023063744661676032 or (T in {7023063744661676032})}]
I20240502 01:59:32.054224 6615 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:59:32.056798 10396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:59:32.057160 10395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:59:32.057520 10398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:59:32.058001 6615 server_base.cc:1034] running on GCE node
I20240502 01:59:32.058269 6615 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:59:32.058323 6615 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:59:32.058352 6615 hybrid_clock.cc:648] HybridClock initialized: now 1714615172058351 us; error 0 us; skew 500 ppm
I20240502 01:59:32.059478 6615 webserver.cc:460] Webserver started at http://127.6.117.196:42229/ using document root <none> and password file <none>
I20240502 01:59:32.059609 6615 fs_manager.cc:359] Metadata directory not provided
I20240502 01:59:32.059649 6615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:59:32.059697 6615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:59:32.060034 6615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714615076189642-6615-0/minicluster-data/ts-3-root/instance:
uuid: "d6ff17402670432da6eeb5da2af7b4b6"
format_stamp: "Formatted at 2024-05-02 01:59:32 on dist-test-slave-89cm"
I20240502 01:59:32.061549 6615 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240502 01:59:32.062511 10403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:59:32.062747 6615 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:32.062814 6615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714615076189642-6615-0/minicluster-data/ts-3-root
uuid: "d6ff17402670432da6eeb5da2af7b4b6"
format_stamp: "Formatted at 2024-05-02 01:59:32 on dist-test-slave-89cm"
I20240502 01:59:32.062903 6615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714615076189642-6615-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714615076189642-6615-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714615076189642-6615-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:59:32.087821 6615 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:59:32.088243 6615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:59:32.088548 6615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:59:32.088990 6615 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:59:32.089030 6615 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:59:32.089064 6615 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:59:32.089098 6615 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:59:32.094044 6615 rpc_server.cc:307] RPC server started. Bound to: 127.6.117.196:34337
I20240502 01:59:32.094095 10466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.117.196:34337 every 8 connection(s)
I20240502 01:59:32.094908 6615 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:59:32.097180 10471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:59:32.097306 10472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:59:32.097683 6615 server_base.cc:1034] running on GCE node
W20240502 01:59:32.098012 10474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:59:32.098248 6615 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:59:32.098304 6615 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:59:32.098326 6615 hybrid_clock.cc:648] HybridClock initialized: now 1714615172098325 us; error 0 us; skew 500 ppm
I20240502 01:59:32.099197 10467 heartbeater.cc:347] Connected to a master server at 127.6.117.254:37527
I20240502 01:59:32.099313 10467 heartbeater.cc:464] Registering TS with master...
I20240502 01:59:32.099372 6615 webserver.cc:460] Webserver started at http://127.6.117.197:36111/ using document root <none> and password file <none>
I20240502 01:59:32.099520 6615 fs_manager.cc:359] Metadata directory not provided
I20240502 01:59:32.099568 10467 heartbeater.cc:511] Master 127.6.117.254:37527 requested a full tablet report, sending...
I20240502 01:59:32.099577 6615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:59:32.099689 6615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:59:32.100077 6615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714615076189642-6615-0/minicluster-data/ts-4-root/instance:
uuid: "d52d70d7578f4995b03ac6faba3ec739"
format_stamp: "Formatted at 2024-05-02 01:59:32 on dist-test-slave-89cm"
I20240502 01:59:32.100206 9947 ts_manager.cc:194] Registered new tserver with Master: d6ff17402670432da6eeb5da2af7b4b6 (127.6.117.196:34337)
I20240502 01:59:32.100782 9947 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43166
I20240502 01:59:32.101536 6615 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:32.102387 10479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:59:32.102644 6615 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:32.102705 6615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714615076189642-6615-0/minicluster-data/ts-4-root
uuid: "d52d70d7578f4995b03ac6faba3ec739"
format_stamp: "Formatted at 2024-05-02 01:59:32 on dist-test-slave-89cm"
I20240502 01:59:32.102762 6615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714615076189642-6615-0/minicluster-data/ts-4-root
metadata directory: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714615076189642-6615-0/minicluster-data/ts-4-root
1 data directories: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714615076189642-6615-0/minicluster-data/ts-4-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:59:32.114145 6615 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:59:32.114562 6615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:59:32.114849 6615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:59:32.115325 6615 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:59:32.115367 6615 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:59:32.115403 6615 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:59:32.115430 6615 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:59:32.119962 6615 rpc_server.cc:307] RPC server started. Bound to: 127.6.117.197:44467
I20240502 01:59:32.120066 10542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.117.197:44467 every 8 connection(s)
I20240502 01:59:32.120846 6615 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:59:32.123243 10546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:59:32.123538 10549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:59:32.123636 10547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:59:32.124229 6615 server_base.cc:1034] running on GCE node
I20240502 01:59:32.124451 6615 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:59:32.124495 6615 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:59:32.124511 6615 hybrid_clock.cc:648] HybridClock initialized: now 1714615172124510 us; error 0 us; skew 500 ppm
I20240502 01:59:32.125418 10543 heartbeater.cc:347] Connected to a master server at 127.6.117.254:37527
I20240502 01:59:32.125519 10543 heartbeater.cc:464] Registering TS with master...
I20240502 01:59:32.125615 6615 webserver.cc:460] Webserver started at http://127.6.117.198:39929/ using document root <none> and password file <none>
I20240502 01:59:32.125769 6615 fs_manager.cc:359] Metadata directory not provided
I20240502 01:59:32.125772 10543 heartbeater.cc:511] Master 127.6.117.254:37527 requested a full tablet report, sending...
I20240502 01:59:32.125823 6615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:59:32.125895 6615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:59:32.126348 6615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714615076189642-6615-0/minicluster-data/ts-5-root/instance:
uuid: "ae04d48c97204162b0f8a27d81fcec1a"
format_stamp: "Formatted at 2024-05-02 01:59:32 on dist-test-slave-89cm"
I20240502 01:59:32.126422 9947 ts_manager.cc:194] Registered new tserver with Master: d52d70d7578f4995b03ac6faba3ec739 (127.6.117.197:44467)
I20240502 01:59:32.127132 9947 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43182
I20240502 01:59:32.128075 6615 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240502 01:59:32.128993 10554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:59:32.129216 6615 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240502 01:59:32.129278 6615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714615076189642-6615-0/minicluster-data/ts-5-root
uuid: "ae04d48c97204162b0f8a27d81fcec1a"
format_stamp: "Formatted at 2024-05-02 01:59:32 on dist-test-slave-89cm"
I20240502 01:59:32.129349 6615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714615076189642-6615-0/minicluster-data/ts-5-root
metadata directory: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714615076189642-6615-0/minicluster-data/ts-5-root
1 data directories: /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714615076189642-6615-0/minicluster-data/ts-5-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:59:32.140219 6615 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:59:32.140707 6615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:59:32.141012 6615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:59:32.141474 6615 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:59:32.141515 6615 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:59:32.141549 6615 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:59:32.141592 6615 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:59:32.146273 6615 rpc_server.cc:307] RPC server started. Bound to: 127.6.117.198:33655
I20240502 01:59:32.146440 10617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.117.198:33655 every 8 connection(s)
I20240502 01:59:32.150559 10618 heartbeater.cc:347] Connected to a master server at 127.6.117.254:37527
I20240502 01:59:32.150676 10618 heartbeater.cc:464] Registering TS with master...
I20240502 01:59:32.150904 10618 heartbeater.cc:511] Master 127.6.117.254:37527 requested a full tablet report, sending...
I20240502 01:59:32.151512 9947 ts_manager.cc:194] Registered new tserver with Master: ae04d48c97204162b0f8a27d81fcec1a (127.6.117.198:33655)
I20240502 01:59:32.152043 9947 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43184
I20240502 01:59:32.902168 10202 consensus_queue.cc:237] T b4409766ce994f698c31aede604a6927 P e77d1c0d41034d1eb5a036dc17c071f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 1.24, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } attrs { replace: true } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "ae04d48c97204162b0f8a27d81fcec1a" member_type: NON_VOTER last_known_addr { host: "127.6.117.198" port: 33655 } attrs { promote: true } }
I20240502 01:59:32.903784 10131 raft_consensus.cc:1261] T b4409766ce994f698c31aede604a6927 P 4b7f7ba2ac084e829c4304c16a94894f [term 1 FOLLOWER]: Refusing update from remote peer e77d1c0d41034d1eb5a036dc17c071f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 1 index: 25. (index mismatch)
I20240502 01:59:32.903795 10053 raft_consensus.cc:1261] T b4409766ce994f698c31aede604a6927 P bdbecaeb8f214529b2012a9c003de2ff [term 1 FOLLOWER]: Refusing update from remote peer e77d1c0d41034d1eb5a036dc17c071f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 1 index: 25. (index mismatch)
I20240502 01:59:32.904179 10623 consensus_queue.cc:1035] T b4409766ce994f698c31aede604a6927 P e77d1c0d41034d1eb5a036dc17c071f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20240502 01:59:32.904415 10264 consensus_queue.cc:1035] T b4409766ce994f698c31aede604a6927 P e77d1c0d41034d1eb5a036dc17c071f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20240502 01:59:32.905891 10624 raft_consensus.cc:2932] T b4409766ce994f698c31aede604a6927 P e77d1c0d41034d1eb5a036dc17c071f5 [term 1 LEADER]: Committing config change with OpId 1.25: config changed from index -1 to 25, NON_VOTER ae04d48c97204162b0f8a27d81fcec1a (127.6.117.198) added. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } attrs { replace: true } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "ae04d48c97204162b0f8a27d81fcec1a" member_type: NON_VOTER last_known_addr { host: "127.6.117.198" port: 33655 } attrs { promote: true } } }
I20240502 01:59:32.906291 10131 raft_consensus.cc:2932] T b4409766ce994f698c31aede604a6927 P 4b7f7ba2ac084e829c4304c16a94894f [term 1 FOLLOWER]: Committing config change with OpId 1.25: config changed from index -1 to 25, NON_VOTER ae04d48c97204162b0f8a27d81fcec1a (127.6.117.198) added. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } attrs { replace: true } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "ae04d48c97204162b0f8a27d81fcec1a" member_type: NON_VOTER last_known_addr { host: "127.6.117.198" port: 33655 } attrs { promote: true } } }
I20240502 01:59:32.906288 10053 raft_consensus.cc:2932] T b4409766ce994f698c31aede604a6927 P bdbecaeb8f214529b2012a9c003de2ff [term 1 FOLLOWER]: Committing config change with OpId 1.25: config changed from index -1 to 25, NON_VOTER ae04d48c97204162b0f8a27d81fcec1a (127.6.117.198) added. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } attrs { replace: true } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "ae04d48c97204162b0f8a27d81fcec1a" member_type: NON_VOTER last_known_addr { host: "127.6.117.198" port: 33655 } attrs { promote: true } } }
W20240502 01:59:32.908567 10172 consensus_peers.cc:485] T b4409766ce994f698c31aede604a6927 P e77d1c0d41034d1eb5a036dc17c071f5 -> Peer ae04d48c97204162b0f8a27d81fcec1a (127.6.117.198:33655): Couldn't send request to peer ae04d48c97204162b0f8a27d81fcec1a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b4409766ce994f698c31aede604a6927. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:32.909065 9947 catalog_manager.cc:5484] T b4409766ce994f698c31aede604a6927 P 4b7f7ba2ac084e829c4304c16a94894f reported cstate change: config changed from index -1 to 25, NON_VOTER ae04d48c97204162b0f8a27d81fcec1a (127.6.117.198) added. New cstate: current_term: 1 leader_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" committed_config { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } attrs { replace: true } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "ae04d48c97204162b0f8a27d81fcec1a" member_type: NON_VOTER last_known_addr { host: "127.6.117.198" port: 33655 } attrs { promote: true } } }
I20240502 01:59:32.913190 10131 consensus_queue.cc:237] T 73ea0bfb40ad43a68a6289ff1b35d005 P 4b7f7ba2ac084e829c4304c16a94894f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 1.24, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "d6ff17402670432da6eeb5da2af7b4b6" member_type: NON_VOTER last_known_addr { host: "127.6.117.196" port: 34337 } attrs { promote: true } }
I20240502 01:59:32.914388 10053 raft_consensus.cc:1261] T 73ea0bfb40ad43a68a6289ff1b35d005 P bdbecaeb8f214529b2012a9c003de2ff [term 1 FOLLOWER]: Refusing update from remote peer 4b7f7ba2ac084e829c4304c16a94894f: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 1 index: 25. (index mismatch)
I20240502 01:59:32.914752 10631 consensus_queue.cc:1035] T 73ea0bfb40ad43a68a6289ff1b35d005 P 4b7f7ba2ac084e829c4304c16a94894f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20240502 01:59:32.914839 10202 raft_consensus.cc:1261] T 73ea0bfb40ad43a68a6289ff1b35d005 P e77d1c0d41034d1eb5a036dc17c071f5 [term 1 FOLLOWER]: Refusing update from remote peer 4b7f7ba2ac084e829c4304c16a94894f: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 1 index: 25. (index mismatch)
I20240502 01:59:32.915191 10631 consensus_queue.cc:1035] T 73ea0bfb40ad43a68a6289ff1b35d005 P 4b7f7ba2ac084e829c4304c16a94894f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20240502 01:59:32.916253 10632 raft_consensus.cc:2932] T 73ea0bfb40ad43a68a6289ff1b35d005 P 4b7f7ba2ac084e829c4304c16a94894f [term 1 LEADER]: Committing config change with OpId 1.25: config changed from index -1 to 25, NON_VOTER d6ff17402670432da6eeb5da2af7b4b6 (127.6.117.196) added. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "d6ff17402670432da6eeb5da2af7b4b6" member_type: NON_VOTER last_known_addr { host: "127.6.117.196" port: 34337 } attrs { promote: true } } }
I20240502 01:59:32.916719 10053 raft_consensus.cc:2932] T 73ea0bfb40ad43a68a6289ff1b35d005 P bdbecaeb8f214529b2012a9c003de2ff [term 1 FOLLOWER]: Committing config change with OpId 1.25: config changed from index -1 to 25, NON_VOTER d6ff17402670432da6eeb5da2af7b4b6 (127.6.117.196) added. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "d6ff17402670432da6eeb5da2af7b4b6" member_type: NON_VOTER last_known_addr { host: "127.6.117.196" port: 34337 } attrs { promote: true } } }
I20240502 01:59:32.918444 10202 raft_consensus.cc:2932] T 73ea0bfb40ad43a68a6289ff1b35d005 P e77d1c0d41034d1eb5a036dc17c071f5 [term 1 FOLLOWER]: Committing config change with OpId 1.25: config changed from index -1 to 25, NON_VOTER d6ff17402670432da6eeb5da2af7b4b6 (127.6.117.196) added. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "d6ff17402670432da6eeb5da2af7b4b6" member_type: NON_VOTER last_known_addr { host: "127.6.117.196" port: 34337 } attrs { promote: true } } }
I20240502 01:59:32.919701 9952 catalog_manager.cc:5484] T 73ea0bfb40ad43a68a6289ff1b35d005 P 4b7f7ba2ac084e829c4304c16a94894f reported cstate change: config changed from index -1 to 25, NON_VOTER d6ff17402670432da6eeb5da2af7b4b6 (127.6.117.196) added. New cstate: current_term: 1 leader_uuid: "4b7f7ba2ac084e829c4304c16a94894f" committed_config { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6ff17402670432da6eeb5da2af7b4b6" member_type: NON_VOTER last_known_addr { host: "127.6.117.196" port: 34337 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 01:59:32.921558 10097 consensus_peers.cc:485] T 73ea0bfb40ad43a68a6289ff1b35d005 P 4b7f7ba2ac084e829c4304c16a94894f -> Peer d6ff17402670432da6eeb5da2af7b4b6 (127.6.117.196:34337): Couldn't send request to peer d6ff17402670432da6eeb5da2af7b4b6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 73ea0bfb40ad43a68a6289ff1b35d005. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:32.923240 10053 consensus_queue.cc:237] T 580db11a61cb46f48d5f7507cfbdbb52 P bdbecaeb8f214529b2012a9c003de2ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } attrs { replace: true } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "d52d70d7578f4995b03ac6faba3ec739" member_type: NON_VOTER last_known_addr { host: "127.6.117.197" port: 44467 } attrs { promote: true } }
I20240502 01:59:32.924470 10131 raft_consensus.cc:1261] T 580db11a61cb46f48d5f7507cfbdbb52 P 4b7f7ba2ac084e829c4304c16a94894f [term 1 FOLLOWER]: Refusing update from remote peer bdbecaeb8f214529b2012a9c003de2ff: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240502 01:59:32.924540 10202 raft_consensus.cc:1261] T 580db11a61cb46f48d5f7507cfbdbb52 P e77d1c0d41034d1eb5a036dc17c071f5 [term 1 FOLLOWER]: Refusing update from remote peer bdbecaeb8f214529b2012a9c003de2ff: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240502 01:59:32.924942 10637 consensus_queue.cc:1035] T 580db11a61cb46f48d5f7507cfbdbb52 P bdbecaeb8f214529b2012a9c003de2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 01:59:32.925518 10638 consensus_queue.cc:1035] T 580db11a61cb46f48d5f7507cfbdbb52 P bdbecaeb8f214529b2012a9c003de2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 01:59:32.926738 10391 raft_consensus.cc:2932] T 580db11a61cb46f48d5f7507cfbdbb52 P bdbecaeb8f214529b2012a9c003de2ff [term 1 LEADER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER d52d70d7578f4995b03ac6faba3ec739 (127.6.117.197) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } attrs { replace: true } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "d52d70d7578f4995b03ac6faba3ec739" member_type: NON_VOTER last_known_addr { host: "127.6.117.197" port: 44467 } attrs { promote: true } } }
I20240502 01:59:32.927093 10202 raft_consensus.cc:2932] T 580db11a61cb46f48d5f7507cfbdbb52 P e77d1c0d41034d1eb5a036dc17c071f5 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER d52d70d7578f4995b03ac6faba3ec739 (127.6.117.197) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } attrs { replace: true } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "d52d70d7578f4995b03ac6faba3ec739" member_type: NON_VOTER last_known_addr { host: "127.6.117.197" port: 44467 } attrs { promote: true } } }
I20240502 01:59:32.927574 10131 raft_consensus.cc:2932] T 580db11a61cb46f48d5f7507cfbdbb52 P 4b7f7ba2ac084e829c4304c16a94894f [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER d52d70d7578f4995b03ac6faba3ec739 (127.6.117.197) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } attrs { replace: true } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "d52d70d7578f4995b03ac6faba3ec739" member_type: NON_VOTER last_known_addr { host: "127.6.117.197" port: 44467 } attrs { promote: true } } }
I20240502 01:59:32.929790 9947 catalog_manager.cc:5484] T 580db11a61cb46f48d5f7507cfbdbb52 P bdbecaeb8f214529b2012a9c003de2ff reported cstate change: config changed from index -1 to 26, NON_VOTER d52d70d7578f4995b03ac6faba3ec739 (127.6.117.197) added. New cstate: current_term: 1 leader_uuid: "bdbecaeb8f214529b2012a9c003de2ff" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d52d70d7578f4995b03ac6faba3ec739" member_type: NON_VOTER last_known_addr { host: "127.6.117.197" port: 44467 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:59:32.930192 10053 consensus_queue.cc:237] T e7a8fc5afcab4b1db4ebf245b66df4be P bdbecaeb8f214529b2012a9c003de2ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } attrs { replace: true } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "ae04d48c97204162b0f8a27d81fcec1a" member_type: NON_VOTER last_known_addr { host: "127.6.117.198" port: 33655 } attrs { promote: true } }
W20240502 01:59:32.931514 10020 consensus_peers.cc:485] T 580db11a61cb46f48d5f7507cfbdbb52 P bdbecaeb8f214529b2012a9c003de2ff -> Peer d52d70d7578f4995b03ac6faba3ec739 (127.6.117.197:44467): Couldn't send request to peer d52d70d7578f4995b03ac6faba3ec739. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 580db11a61cb46f48d5f7507cfbdbb52. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:32.931757 10202 raft_consensus.cc:1261] T e7a8fc5afcab4b1db4ebf245b66df4be P e77d1c0d41034d1eb5a036dc17c071f5 [term 1 FOLLOWER]: Refusing update from remote peer bdbecaeb8f214529b2012a9c003de2ff: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240502 01:59:32.932080 10638 consensus_queue.cc:1035] T e7a8fc5afcab4b1db4ebf245b66df4be P bdbecaeb8f214529b2012a9c003de2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 01:59:32.933009 10131 raft_consensus.cc:1261] T e7a8fc5afcab4b1db4ebf245b66df4be P 4b7f7ba2ac084e829c4304c16a94894f [term 1 FOLLOWER]: Refusing update from remote peer bdbecaeb8f214529b2012a9c003de2ff: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240502 01:59:32.933573 10391 consensus_queue.cc:1035] T e7a8fc5afcab4b1db4ebf245b66df4be P bdbecaeb8f214529b2012a9c003de2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 01:59:32.933859 10638 raft_consensus.cc:2932] T e7a8fc5afcab4b1db4ebf245b66df4be P bdbecaeb8f214529b2012a9c003de2ff [term 1 LEADER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER ae04d48c97204162b0f8a27d81fcec1a (127.6.117.198) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } attrs { replace: true } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "ae04d48c97204162b0f8a27d81fcec1a" member_type: NON_VOTER last_known_addr { host: "127.6.117.198" port: 33655 } attrs { promote: true } } }
I20240502 01:59:32.934234 10202 raft_consensus.cc:2932] T e7a8fc5afcab4b1db4ebf245b66df4be P e77d1c0d41034d1eb5a036dc17c071f5 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER ae04d48c97204162b0f8a27d81fcec1a (127.6.117.198) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } attrs { replace: true } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "ae04d48c97204162b0f8a27d81fcec1a" member_type: NON_VOTER last_known_addr { host: "127.6.117.198" port: 33655 } attrs { promote: true } } }
W20240502 01:59:32.935384 10021 consensus_peers.cc:485] T e7a8fc5afcab4b1db4ebf245b66df4be P bdbecaeb8f214529b2012a9c003de2ff -> Peer ae04d48c97204162b0f8a27d81fcec1a (127.6.117.198:33655): Couldn't send request to peer ae04d48c97204162b0f8a27d81fcec1a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e7a8fc5afcab4b1db4ebf245b66df4be. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:32.935556 10131 raft_consensus.cc:2932] T e7a8fc5afcab4b1db4ebf245b66df4be P 4b7f7ba2ac084e829c4304c16a94894f [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER ae04d48c97204162b0f8a27d81fcec1a (127.6.117.198) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } attrs { replace: true } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "ae04d48c97204162b0f8a27d81fcec1a" member_type: NON_VOTER last_known_addr { host: "127.6.117.198" port: 33655 } attrs { promote: true } } }
I20240502 01:59:32.936471 9953 catalog_manager.cc:5484] T e7a8fc5afcab4b1db4ebf245b66df4be P e77d1c0d41034d1eb5a036dc17c071f5 reported cstate change: config changed from index -1 to 26, NON_VOTER ae04d48c97204162b0f8a27d81fcec1a (127.6.117.198) added. New cstate: current_term: 1 leader_uuid: "bdbecaeb8f214529b2012a9c003de2ff" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } attrs { replace: true } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "ae04d48c97204162b0f8a27d81fcec1a" member_type: NON_VOTER last_known_addr { host: "127.6.117.198" port: 33655 } attrs { promote: true } } }
I20240502 01:59:32.938954 10131 consensus_queue.cc:237] T 19ff9c216a094fde86ea9f9648bf08cb P 4b7f7ba2ac084e829c4304c16a94894f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 1.24, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } attrs { replace: true } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "d52d70d7578f4995b03ac6faba3ec739" member_type: NON_VOTER last_known_addr { host: "127.6.117.197" port: 44467 } attrs { promote: true } }
I20240502 01:59:32.940322 10053 raft_consensus.cc:1261] T 19ff9c216a094fde86ea9f9648bf08cb P bdbecaeb8f214529b2012a9c003de2ff [term 1 FOLLOWER]: Refusing update from remote peer 4b7f7ba2ac084e829c4304c16a94894f: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 1 index: 25. (index mismatch)
I20240502 01:59:32.940675 10393 consensus_queue.cc:1035] T 19ff9c216a094fde86ea9f9648bf08cb P 4b7f7ba2ac084e829c4304c16a94894f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20240502 01:59:32.941017 10202 raft_consensus.cc:1261] T 19ff9c216a094fde86ea9f9648bf08cb P e77d1c0d41034d1eb5a036dc17c071f5 [term 1 FOLLOWER]: Refusing update from remote peer 4b7f7ba2ac084e829c4304c16a94894f: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 1 index: 25. (index mismatch)
I20240502 01:59:32.941332 10393 consensus_queue.cc:1035] T 19ff9c216a094fde86ea9f9648bf08cb P 4b7f7ba2ac084e829c4304c16a94894f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20240502 01:59:32.942112 10631 raft_consensus.cc:2932] T 19ff9c216a094fde86ea9f9648bf08cb P 4b7f7ba2ac084e829c4304c16a94894f [term 1 LEADER]: Committing config change with OpId 1.25: config changed from index -1 to 25, NON_VOTER d52d70d7578f4995b03ac6faba3ec739 (127.6.117.197) added. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } attrs { replace: true } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "d52d70d7578f4995b03ac6faba3ec739" member_type: NON_VOTER last_known_addr { host: "127.6.117.197" port: 44467 } attrs { promote: true } } }
I20240502 01:59:32.942456 10053 raft_consensus.cc:2932] T 19ff9c216a094fde86ea9f9648bf08cb P bdbecaeb8f214529b2012a9c003de2ff [term 1 FOLLOWER]: Committing config change with OpId 1.25: config changed from index -1 to 25, NON_VOTER d52d70d7578f4995b03ac6faba3ec739 (127.6.117.197) added. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } attrs { replace: true } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "d52d70d7578f4995b03ac6faba3ec739" member_type: NON_VOTER last_known_addr { host: "127.6.117.197" port: 44467 } attrs { promote: true } } }
W20240502 01:59:32.944408 10096 consensus_peers.cc:485] T 19ff9c216a094fde86ea9f9648bf08cb P 4b7f7ba2ac084e829c4304c16a94894f -> Peer d52d70d7578f4995b03ac6faba3ec739 (127.6.117.197:44467): Couldn't send request to peer d52d70d7578f4995b03ac6faba3ec739. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 19ff9c216a094fde86ea9f9648bf08cb. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:32.944653 10200 raft_consensus.cc:2932] T 19ff9c216a094fde86ea9f9648bf08cb P e77d1c0d41034d1eb5a036dc17c071f5 [term 1 FOLLOWER]: Committing config change with OpId 1.25: config changed from index -1 to 25, NON_VOTER d52d70d7578f4995b03ac6faba3ec739 (127.6.117.197) added. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } attrs { replace: true } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "d52d70d7578f4995b03ac6faba3ec739" member_type: NON_VOTER last_known_addr { host: "127.6.117.197" port: 44467 } attrs { promote: true } } }
I20240502 01:59:32.944830 9952 catalog_manager.cc:5484] T 19ff9c216a094fde86ea9f9648bf08cb P 4b7f7ba2ac084e829c4304c16a94894f reported cstate change: config changed from index -1 to 25, NON_VOTER d52d70d7578f4995b03ac6faba3ec739 (127.6.117.197) added. New cstate: current_term: 1 leader_uuid: "4b7f7ba2ac084e829c4304c16a94894f" committed_config { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d52d70d7578f4995b03ac6faba3ec739" member_type: NON_VOTER last_known_addr { host: "127.6.117.197" port: 44467 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:59:32.946388 10131 consensus_queue.cc:237] T 1851ea6467584b0ea32a1f68696803d0 P 4b7f7ba2ac084e829c4304c16a94894f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "d6ff17402670432da6eeb5da2af7b4b6" member_type: NON_VOTER last_known_addr { host: "127.6.117.196" port: 34337 } attrs { promote: true } }
I20240502 01:59:32.947634 10053 raft_consensus.cc:1261] T 1851ea6467584b0ea32a1f68696803d0 P bdbecaeb8f214529b2012a9c003de2ff [term 1 FOLLOWER]: Refusing update from remote peer 4b7f7ba2ac084e829c4304c16a94894f: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
W20240502 01:59:32.947739 10097 consensus_peers.cc:485] T 1851ea6467584b0ea32a1f68696803d0 P 4b7f7ba2ac084e829c4304c16a94894f -> Peer d6ff17402670432da6eeb5da2af7b4b6 (127.6.117.196:34337): Couldn't send request to peer d6ff17402670432da6eeb5da2af7b4b6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1851ea6467584b0ea32a1f68696803d0. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:32.947976 10393 consensus_queue.cc:1035] T 1851ea6467584b0ea32a1f68696803d0 P 4b7f7ba2ac084e829c4304c16a94894f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 01:59:32.948058 10200 raft_consensus.cc:1261] T 1851ea6467584b0ea32a1f68696803d0 P e77d1c0d41034d1eb5a036dc17c071f5 [term 1 FOLLOWER]: Refusing update from remote peer 4b7f7ba2ac084e829c4304c16a94894f: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240502 01:59:32.948408 10393 consensus_queue.cc:1035] T 1851ea6467584b0ea32a1f68696803d0 P 4b7f7ba2ac084e829c4304c16a94894f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 01:59:32.949474 10631 raft_consensus.cc:2932] T 1851ea6467584b0ea32a1f68696803d0 P 4b7f7ba2ac084e829c4304c16a94894f [term 1 LEADER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER d6ff17402670432da6eeb5da2af7b4b6 (127.6.117.196) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "d6ff17402670432da6eeb5da2af7b4b6" member_type: NON_VOTER last_known_addr { host: "127.6.117.196" port: 34337 } attrs { promote: true } } }
I20240502 01:59:32.949817 10053 raft_consensus.cc:2932] T 1851ea6467584b0ea32a1f68696803d0 P bdbecaeb8f214529b2012a9c003de2ff [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER d6ff17402670432da6eeb5da2af7b4b6 (127.6.117.196) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "d6ff17402670432da6eeb5da2af7b4b6" member_type: NON_VOTER last_known_addr { host: "127.6.117.196" port: 34337 } attrs { promote: true } } }
I20240502 01:59:32.950148 10200 raft_consensus.cc:2932] T 1851ea6467584b0ea32a1f68696803d0 P e77d1c0d41034d1eb5a036dc17c071f5 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER d6ff17402670432da6eeb5da2af7b4b6 (127.6.117.196) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "d6ff17402670432da6eeb5da2af7b4b6" member_type: NON_VOTER last_known_addr { host: "127.6.117.196" port: 34337 } attrs { promote: true } } }
I20240502 01:59:32.951973 9947 catalog_manager.cc:5484] T 1851ea6467584b0ea32a1f68696803d0 P 4b7f7ba2ac084e829c4304c16a94894f reported cstate change: config changed from index -1 to 26, NON_VOTER d6ff17402670432da6eeb5da2af7b4b6 (127.6.117.196) added. New cstate: current_term: 1 leader_uuid: "4b7f7ba2ac084e829c4304c16a94894f" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6ff17402670432da6eeb5da2af7b4b6" member_type: NON_VOTER last_known_addr { host: "127.6.117.196" port: 34337 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:59:33.101660 10467 heartbeater.cc:503] Master 127.6.117.254:37527 was elected leader, sending a full tablet report...
I20240502 01:59:33.127969 10543 heartbeater.cc:503] Master 127.6.117.254:37527 was elected leader, sending a full tablet report...
I20240502 01:59:33.152962 10618 heartbeater.cc:503] Master 127.6.117.254:37527 was elected leader, sending a full tablet report...
I20240502 01:59:33.310626 10642 ts_tablet_manager.cc:902] T b4409766ce994f698c31aede604a6927 P ae04d48c97204162b0f8a27d81fcec1a: Initiating tablet copy from peer e77d1c0d41034d1eb5a036dc17c071f5 (127.6.117.195:43977)
I20240502 01:59:33.310882 10643 ts_tablet_manager.cc:902] T 73ea0bfb40ad43a68a6289ff1b35d005 P d6ff17402670432da6eeb5da2af7b4b6: Initiating tablet copy from peer 4b7f7ba2ac084e829c4304c16a94894f (127.6.117.194:46805)
I20240502 01:59:33.311228 10642 tablet_copy_client.cc:308] T b4409766ce994f698c31aede604a6927 P ae04d48c97204162b0f8a27d81fcec1a: tablet copy: Beginning tablet copy session from remote peer at address 127.6.117.195:43977
I20240502 01:59:33.311523 10643 tablet_copy_client.cc:308] T 73ea0bfb40ad43a68a6289ff1b35d005 P d6ff17402670432da6eeb5da2af7b4b6: tablet copy: Beginning tablet copy session from remote peer at address 127.6.117.194:46805
I20240502 01:59:33.315181 10218 tablet_copy_service.cc:140] P e77d1c0d41034d1eb5a036dc17c071f5: Received BeginTabletCopySession request for tablet b4409766ce994f698c31aede604a6927 from peer ae04d48c97204162b0f8a27d81fcec1a ({username='slave'} at 127.0.0.1:48102)
I20240502 01:59:33.315289 10218 tablet_copy_service.cc:161] P e77d1c0d41034d1eb5a036dc17c071f5: Beginning new tablet copy session on tablet b4409766ce994f698c31aede604a6927 from peer ae04d48c97204162b0f8a27d81fcec1a at {username='slave'} at 127.0.0.1:48102: session id = ae04d48c97204162b0f8a27d81fcec1a-b4409766ce994f698c31aede604a6927
I20240502 01:59:33.316535 10218 tablet_copy_source_session.cc:205] T b4409766ce994f698c31aede604a6927 P e77d1c0d41034d1eb5a036dc17c071f5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:59:33.316663 10143 tablet_copy_service.cc:140] P 4b7f7ba2ac084e829c4304c16a94894f: Received BeginTabletCopySession request for tablet 73ea0bfb40ad43a68a6289ff1b35d005 from peer d6ff17402670432da6eeb5da2af7b4b6 ({username='slave'} at 127.0.0.1:52658)
I20240502 01:59:33.316754 10143 tablet_copy_service.cc:161] P 4b7f7ba2ac084e829c4304c16a94894f: Beginning new tablet copy session on tablet 73ea0bfb40ad43a68a6289ff1b35d005 from peer d6ff17402670432da6eeb5da2af7b4b6 at {username='slave'} at 127.0.0.1:52658: session id = d6ff17402670432da6eeb5da2af7b4b6-73ea0bfb40ad43a68a6289ff1b35d005
I20240502 01:59:33.317436 10642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4409766ce994f698c31aede604a6927. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:33.317633 10143 tablet_copy_source_session.cc:205] T 73ea0bfb40ad43a68a6289ff1b35d005 P 4b7f7ba2ac084e829c4304c16a94894f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:59:33.318385 10643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73ea0bfb40ad43a68a6289ff1b35d005. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:33.319600 10642 tablet_copy_client.cc:783] T b4409766ce994f698c31aede604a6927 P ae04d48c97204162b0f8a27d81fcec1a: tablet copy: Starting download of 0 data blocks...
I20240502 01:59:33.319739 10642 tablet_copy_client.cc:647] T b4409766ce994f698c31aede604a6927 P ae04d48c97204162b0f8a27d81fcec1a: tablet copy: Starting download of 1 WAL segments...
I20240502 01:59:33.320325 10643 tablet_copy_client.cc:783] T 73ea0bfb40ad43a68a6289ff1b35d005 P d6ff17402670432da6eeb5da2af7b4b6: tablet copy: Starting download of 0 data blocks...
I20240502 01:59:33.320484 10643 tablet_copy_client.cc:647] T 73ea0bfb40ad43a68a6289ff1b35d005 P d6ff17402670432da6eeb5da2af7b4b6: tablet copy: Starting download of 1 WAL segments...
I20240502 01:59:33.320720 10647 tablet_copy_client.cc:1002] T b4409766ce994f698c31aede604a6927 P ae04d48c97204162b0f8a27d81fcec1a: tablet copy: Injecting latency into file download: 2000
I20240502 01:59:33.321439 10650 tablet_copy_client.cc:1002] T 73ea0bfb40ad43a68a6289ff1b35d005 P d6ff17402670432da6eeb5da2af7b4b6: tablet copy: Injecting latency into file download: 2000
I20240502 01:59:33.429128 10654 ts_tablet_manager.cc:902] T 580db11a61cb46f48d5f7507cfbdbb52 P d52d70d7578f4995b03ac6faba3ec739: Initiating tablet copy from peer bdbecaeb8f214529b2012a9c003de2ff (127.6.117.193:34597)
I20240502 01:59:33.429831 10654 tablet_copy_client.cc:308] T 580db11a61cb46f48d5f7507cfbdbb52 P d52d70d7578f4995b03ac6faba3ec739: tablet copy: Beginning tablet copy session from remote peer at address 127.6.117.193:34597
I20240502 01:59:33.433940 10067 tablet_copy_service.cc:140] P bdbecaeb8f214529b2012a9c003de2ff: Received BeginTabletCopySession request for tablet 580db11a61cb46f48d5f7507cfbdbb52 from peer d52d70d7578f4995b03ac6faba3ec739 ({username='slave'} at 127.0.0.1:44448)
I20240502 01:59:33.434064 10067 tablet_copy_service.cc:161] P bdbecaeb8f214529b2012a9c003de2ff: Beginning new tablet copy session on tablet 580db11a61cb46f48d5f7507cfbdbb52 from peer d52d70d7578f4995b03ac6faba3ec739 at {username='slave'} at 127.0.0.1:44448: session id = d52d70d7578f4995b03ac6faba3ec739-580db11a61cb46f48d5f7507cfbdbb52
I20240502 01:59:33.435005 10067 tablet_copy_source_session.cc:205] T 580db11a61cb46f48d5f7507cfbdbb52 P bdbecaeb8f214529b2012a9c003de2ff: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:59:33.435827 10654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 580db11a61cb46f48d5f7507cfbdbb52. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:33.437703 10654 tablet_copy_client.cc:783] T 580db11a61cb46f48d5f7507cfbdbb52 P d52d70d7578f4995b03ac6faba3ec739: tablet copy: Starting download of 0 data blocks...
I20240502 01:59:33.437850 10654 tablet_copy_client.cc:647] T 580db11a61cb46f48d5f7507cfbdbb52 P d52d70d7578f4995b03ac6faba3ec739: tablet copy: Starting download of 1 WAL segments...
I20240502 01:59:33.438709 10658 tablet_copy_client.cc:1002] T 580db11a61cb46f48d5f7507cfbdbb52 P d52d70d7578f4995b03ac6faba3ec739: tablet copy: Injecting latency into file download: 2000
I20240502 01:59:33.506455 10661 ts_tablet_manager.cc:902] T e7a8fc5afcab4b1db4ebf245b66df4be P ae04d48c97204162b0f8a27d81fcec1a: Initiating tablet copy from peer bdbecaeb8f214529b2012a9c003de2ff (127.6.117.193:34597)
I20240502 01:59:33.507005 10661 tablet_copy_client.cc:308] T e7a8fc5afcab4b1db4ebf245b66df4be P ae04d48c97204162b0f8a27d81fcec1a: tablet copy: Beginning tablet copy session from remote peer at address 127.6.117.193:34597
I20240502 01:59:33.510331 10067 tablet_copy_service.cc:140] P bdbecaeb8f214529b2012a9c003de2ff: Received BeginTabletCopySession request for tablet e7a8fc5afcab4b1db4ebf245b66df4be from peer ae04d48c97204162b0f8a27d81fcec1a ({username='slave'} at 127.0.0.1:44462)
I20240502 01:59:33.510432 10067 tablet_copy_service.cc:161] P bdbecaeb8f214529b2012a9c003de2ff: Beginning new tablet copy session on tablet e7a8fc5afcab4b1db4ebf245b66df4be from peer ae04d48c97204162b0f8a27d81fcec1a at {username='slave'} at 127.0.0.1:44462: session id = ae04d48c97204162b0f8a27d81fcec1a-e7a8fc5afcab4b1db4ebf245b66df4be
I20240502 01:59:33.511211 10067 tablet_copy_source_session.cc:205] T e7a8fc5afcab4b1db4ebf245b66df4be P bdbecaeb8f214529b2012a9c003de2ff: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:59:33.511933 10661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7a8fc5afcab4b1db4ebf245b66df4be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:33.514047 10661 tablet_copy_client.cc:783] T e7a8fc5afcab4b1db4ebf245b66df4be P ae04d48c97204162b0f8a27d81fcec1a: tablet copy: Starting download of 0 data blocks...
I20240502 01:59:33.514199 10661 tablet_copy_client.cc:647] T e7a8fc5afcab4b1db4ebf245b66df4be P ae04d48c97204162b0f8a27d81fcec1a: tablet copy: Starting download of 1 WAL segments...
I20240502 01:59:33.515130 10665 tablet_copy_client.cc:1002] T e7a8fc5afcab4b1db4ebf245b66df4be P ae04d48c97204162b0f8a27d81fcec1a: tablet copy: Injecting latency into file download: 2000
I20240502 01:59:33.548352 10666 ts_tablet_manager.cc:902] T 1851ea6467584b0ea32a1f68696803d0 P d6ff17402670432da6eeb5da2af7b4b6: Initiating tablet copy from peer 4b7f7ba2ac084e829c4304c16a94894f (127.6.117.194:46805)
I20240502 01:59:33.549023 10666 tablet_copy_client.cc:308] T 1851ea6467584b0ea32a1f68696803d0 P d6ff17402670432da6eeb5da2af7b4b6: tablet copy: Beginning tablet copy session from remote peer at address 127.6.117.194:46805
I20240502 01:59:33.549436 10143 tablet_copy_service.cc:140] P 4b7f7ba2ac084e829c4304c16a94894f: Received BeginTabletCopySession request for tablet 1851ea6467584b0ea32a1f68696803d0 from peer d6ff17402670432da6eeb5da2af7b4b6 ({username='slave'} at 127.0.0.1:52658)
I20240502 01:59:33.549527 10143 tablet_copy_service.cc:161] P 4b7f7ba2ac084e829c4304c16a94894f: Beginning new tablet copy session on tablet 1851ea6467584b0ea32a1f68696803d0 from peer d6ff17402670432da6eeb5da2af7b4b6 at {username='slave'} at 127.0.0.1:52658: session id = d6ff17402670432da6eeb5da2af7b4b6-1851ea6467584b0ea32a1f68696803d0
I20240502 01:59:33.550621 10143 tablet_copy_source_session.cc:205] T 1851ea6467584b0ea32a1f68696803d0 P 4b7f7ba2ac084e829c4304c16a94894f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:59:33.551373 10666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1851ea6467584b0ea32a1f68696803d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:33.553588 10666 tablet_copy_client.cc:783] T 1851ea6467584b0ea32a1f68696803d0 P d6ff17402670432da6eeb5da2af7b4b6: tablet copy: Starting download of 0 data blocks...
I20240502 01:59:33.553746 10666 tablet_copy_client.cc:647] T 1851ea6467584b0ea32a1f68696803d0 P d6ff17402670432da6eeb5da2af7b4b6: tablet copy: Starting download of 1 WAL segments...
I20240502 01:59:33.554643 10670 tablet_copy_client.cc:1002] T 1851ea6467584b0ea32a1f68696803d0 P d6ff17402670432da6eeb5da2af7b4b6: tablet copy: Injecting latency into file download: 2000
I20240502 01:59:33.564014 10671 ts_tablet_manager.cc:902] T 19ff9c216a094fde86ea9f9648bf08cb P d52d70d7578f4995b03ac6faba3ec739: Initiating tablet copy from peer 4b7f7ba2ac084e829c4304c16a94894f (127.6.117.194:46805)
I20240502 01:59:33.564653 10671 tablet_copy_client.cc:308] T 19ff9c216a094fde86ea9f9648bf08cb P d52d70d7578f4995b03ac6faba3ec739: tablet copy: Beginning tablet copy session from remote peer at address 127.6.117.194:46805
I20240502 01:59:33.567761 10143 tablet_copy_service.cc:140] P 4b7f7ba2ac084e829c4304c16a94894f: Received BeginTabletCopySession request for tablet 19ff9c216a094fde86ea9f9648bf08cb from peer d52d70d7578f4995b03ac6faba3ec739 ({username='slave'} at 127.0.0.1:52672)
I20240502 01:59:33.567867 10143 tablet_copy_service.cc:161] P 4b7f7ba2ac084e829c4304c16a94894f: Beginning new tablet copy session on tablet 19ff9c216a094fde86ea9f9648bf08cb from peer d52d70d7578f4995b03ac6faba3ec739 at {username='slave'} at 127.0.0.1:52672: session id = d52d70d7578f4995b03ac6faba3ec739-19ff9c216a094fde86ea9f9648bf08cb
I20240502 01:59:33.568913 10143 tablet_copy_source_session.cc:205] T 19ff9c216a094fde86ea9f9648bf08cb P 4b7f7ba2ac084e829c4304c16a94894f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:59:33.569710 10671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19ff9c216a094fde86ea9f9648bf08cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:33.571870 10671 tablet_copy_client.cc:783] T 19ff9c216a094fde86ea9f9648bf08cb P d52d70d7578f4995b03ac6faba3ec739: tablet copy: Starting download of 0 data blocks...
I20240502 01:59:33.572064 10671 tablet_copy_client.cc:647] T 19ff9c216a094fde86ea9f9648bf08cb P d52d70d7578f4995b03ac6faba3ec739: tablet copy: Starting download of 1 WAL segments...
I20240502 01:59:33.573055 10675 tablet_copy_client.cc:1002] T 19ff9c216a094fde86ea9f9648bf08cb P d52d70d7578f4995b03ac6faba3ec739: tablet copy: Injecting latency into file download: 2000
I20240502 01:59:33.953724 10008 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:59:33.959200 10200 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b4409766ce994f698c31aede604a6927"
dest_uuid: "e77d1c0d41034d1eb5a036dc17c071f5"
mode: GRACEFUL
new_leader_uuid: "bdbecaeb8f214529b2012a9c003de2ff"
from {username='slave'} at 127.0.0.1:48104
I20240502 01:59:33.959355 10200 raft_consensus.cc:595] T b4409766ce994f698c31aede604a6927 P e77d1c0d41034d1eb5a036dc17c071f5 [term 1 LEADER]: Received request to transfer leadership to TS bdbecaeb8f214529b2012a9c003de2ff
I20240502 01:59:33.960101 10054 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e7a8fc5afcab4b1db4ebf245b66df4be"
dest_uuid: "bdbecaeb8f214529b2012a9c003de2ff"
mode: GRACEFUL
new_leader_uuid: "4b7f7ba2ac084e829c4304c16a94894f"
from {username='slave'} at 127.0.0.1:44420
I20240502 01:59:33.960206 10054 raft_consensus.cc:595] T e7a8fc5afcab4b1db4ebf245b66df4be P bdbecaeb8f214529b2012a9c003de2ff [term 1 LEADER]: Received request to transfer leadership to TS 4b7f7ba2ac084e829c4304c16a94894f
I20240502 01:59:33.960506 10008 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 2
I20240502 01:59:33.960636 10008 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:59:34.008379 10678 raft_consensus.cc:979] T e7a8fc5afcab4b1db4ebf245b66df4be P bdbecaeb8f214529b2012a9c003de2ff: : Instructing follower 4b7f7ba2ac084e829c4304c16a94894f to start an election
I20240502 01:59:34.008517 10678 raft_consensus.cc:1067] T e7a8fc5afcab4b1db4ebf245b66df4be P bdbecaeb8f214529b2012a9c003de2ff [term 1 LEADER]: Signalling peer 4b7f7ba2ac084e829c4304c16a94894f to start an election
I20240502 01:59:34.008900 10129 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e7a8fc5afcab4b1db4ebf245b66df4be"
dest_uuid: "4b7f7ba2ac084e829c4304c16a94894f"
from {username='slave'} at 127.0.0.1:52602
I20240502 01:59:34.009024 10129 raft_consensus.cc:482] T e7a8fc5afcab4b1db4ebf245b66df4be P 4b7f7ba2ac084e829c4304c16a94894f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:59:34.009114 10129 raft_consensus.cc:3037] T e7a8fc5afcab4b1db4ebf245b66df4be P 4b7f7ba2ac084e829c4304c16a94894f [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:59:34.010259 10129 raft_consensus.cc:504] T e7a8fc5afcab4b1db4ebf245b66df4be P 4b7f7ba2ac084e829c4304c16a94894f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } attrs { replace: true } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "ae04d48c97204162b0f8a27d81fcec1a" member_type: NON_VOTER last_known_addr { host: "127.6.117.198" port: 33655 } attrs { promote: true } }
I20240502 01:59:34.010749 10129 leader_election.cc:288] T e7a8fc5afcab4b1db4ebf245b66df4be P 4b7f7ba2ac084e829c4304c16a94894f [CANDIDATE]: Term 2 election: Requested vote from peers bdbecaeb8f214529b2012a9c003de2ff (127.6.117.193:34597), e77d1c0d41034d1eb5a036dc17c071f5 (127.6.117.195:43977)
I20240502 01:59:34.010912 10200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7a8fc5afcab4b1db4ebf245b66df4be" candidate_uuid: "4b7f7ba2ac084e829c4304c16a94894f" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: true dest_uuid: "e77d1c0d41034d1eb5a036dc17c071f5"
I20240502 01:59:34.011111 10200 raft_consensus.cc:3037] T e7a8fc5afcab4b1db4ebf245b66df4be P e77d1c0d41034d1eb5a036dc17c071f5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:59:34.010941 10054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7a8fc5afcab4b1db4ebf245b66df4be" candidate_uuid: "4b7f7ba2ac084e829c4304c16a94894f" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: true dest_uuid: "bdbecaeb8f214529b2012a9c003de2ff"
I20240502 01:59:34.011201 10054 raft_consensus.cc:3032] T e7a8fc5afcab4b1db4ebf245b66df4be P bdbecaeb8f214529b2012a9c003de2ff [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:59:34.011241 10054 raft_consensus.cc:729] T e7a8fc5afcab4b1db4ebf245b66df4be P bdbecaeb8f214529b2012a9c003de2ff [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bdbecaeb8f214529b2012a9c003de2ff, State: Running, Role: LEADER
I20240502 01:59:34.011384 10054 consensus_queue.cc:260] T e7a8fc5afcab4b1db4ebf245b66df4be P bdbecaeb8f214529b2012a9c003de2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } attrs { replace: true } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "ae04d48c97204162b0f8a27d81fcec1a" member_type: NON_VOTER last_known_addr { host: "127.6.117.198" port: 33655 } attrs { promote: true } }
I20240502 01:59:34.011559 10054 raft_consensus.cc:3037] T e7a8fc5afcab4b1db4ebf245b66df4be P bdbecaeb8f214529b2012a9c003de2ff [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:59:34.012017 10200 raft_consensus.cc:2454] T e7a8fc5afcab4b1db4ebf245b66df4be P e77d1c0d41034d1eb5a036dc17c071f5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b7f7ba2ac084e829c4304c16a94894f in term 2.
I20240502 01:59:34.012337 10097 leader_election.cc:302] T e7a8fc5afcab4b1db4ebf245b66df4be P 4b7f7ba2ac084e829c4304c16a94894f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b7f7ba2ac084e829c4304c16a94894f, e77d1c0d41034d1eb5a036dc17c071f5; no voters:
I20240502 01:59:34.012460 10054 raft_consensus.cc:2454] T e7a8fc5afcab4b1db4ebf245b66df4be P bdbecaeb8f214529b2012a9c003de2ff [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b7f7ba2ac084e829c4304c16a94894f in term 2.
I20240502 01:59:34.012523 10393 raft_consensus.cc:2781] T e7a8fc5afcab4b1db4ebf245b66df4be P 4b7f7ba2ac084e829c4304c16a94894f [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:59:34.012596 10393 raft_consensus.cc:686] T e7a8fc5afcab4b1db4ebf245b66df4be P 4b7f7ba2ac084e829c4304c16a94894f [term 2 LEADER]: Becoming Leader. State: Replica: 4b7f7ba2ac084e829c4304c16a94894f, State: Running, Role: LEADER
I20240502 01:59:34.012717 10393 consensus_queue.cc:237] T e7a8fc5afcab4b1db4ebf245b66df4be P 4b7f7ba2ac084e829c4304c16a94894f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } attrs { replace: true } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "ae04d48c97204162b0f8a27d81fcec1a" member_type: NON_VOTER last_known_addr { host: "127.6.117.198" port: 33655 } attrs { promote: true } }
I20240502 01:59:34.014500 9952 catalog_manager.cc:5484] T e7a8fc5afcab4b1db4ebf245b66df4be P 4b7f7ba2ac084e829c4304c16a94894f reported cstate change: term changed from 1 to 2, leader changed from bdbecaeb8f214529b2012a9c003de2ff (127.6.117.193) to 4b7f7ba2ac084e829c4304c16a94894f (127.6.117.194). New cstate: current_term: 2 leader_uuid: "4b7f7ba2ac084e829c4304c16a94894f" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae04d48c97204162b0f8a27d81fcec1a" member_type: NON_VOTER last_known_addr { host: "127.6.117.198" port: 33655 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:59:34.020417 10129 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e7a8fc5afcab4b1db4ebf245b66df4be"
dest_uuid: "4b7f7ba2ac084e829c4304c16a94894f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52598
I20240502 01:59:34.020576 10129 raft_consensus.cc:595] T e7a8fc5afcab4b1db4ebf245b66df4be P 4b7f7ba2ac084e829c4304c16a94894f [term 2 LEADER]: Received request to transfer leadership
I20240502 01:59:34.025020 10129 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e7a8fc5afcab4b1db4ebf245b66df4be"
dest_uuid: "4b7f7ba2ac084e829c4304c16a94894f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52598
I20240502 01:59:34.025178 10129 raft_consensus.cc:595] T e7a8fc5afcab4b1db4ebf245b66df4be P 4b7f7ba2ac084e829c4304c16a94894f [term 2 LEADER]: Received request to transfer leadership
W20240502 01:59:34.025454 9940 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: leadership transfer for e7a8fc5afcab4b1db4ebf245b66df4be already in progress [suppressed 3 similar messages]
I20240502 01:59:34.026106 10129 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e7a8fc5afcab4b1db4ebf245b66df4be"
dest_uuid: "4b7f7ba2ac084e829c4304c16a94894f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52598
I20240502 01:59:34.026213 10129 raft_consensus.cc:595] T e7a8fc5afcab4b1db4ebf245b66df4be P 4b7f7ba2ac084e829c4304c16a94894f [term 2 LEADER]: Received request to transfer leadership
W20240502 01:59:34.026526 10003 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for e7a8fc5afcab4b1db4ebf245b66df4be already in progress
W20240502 01:59:34.026587 10003 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for e7a8fc5afcab4b1db4ebf245b66df4be already in progress
I20240502 01:59:34.200536 10680 raft_consensus.cc:979] T b4409766ce994f698c31aede604a6927 P e77d1c0d41034d1eb5a036dc17c071f5: : Instructing follower bdbecaeb8f214529b2012a9c003de2ff to start an election
I20240502 01:59:34.200693 10681 raft_consensus.cc:1067] T b4409766ce994f698c31aede604a6927 P e77d1c0d41034d1eb5a036dc17c071f5 [term 1 LEADER]: Signalling peer bdbecaeb8f214529b2012a9c003de2ff to start an election
I20240502 01:59:34.201128 10053 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b4409766ce994f698c31aede604a6927"
dest_uuid: "bdbecaeb8f214529b2012a9c003de2ff"
from {username='slave'} at 127.0.0.1:44398
I20240502 01:59:34.201254 10053 raft_consensus.cc:482] T b4409766ce994f698c31aede604a6927 P bdbecaeb8f214529b2012a9c003de2ff [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:59:34.201316 10053 raft_consensus.cc:3037] T b4409766ce994f698c31aede604a6927 P bdbecaeb8f214529b2012a9c003de2ff [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:59:34.202147 10053 raft_consensus.cc:504] T b4409766ce994f698c31aede604a6927 P bdbecaeb8f214529b2012a9c003de2ff [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } attrs { replace: true } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "ae04d48c97204162b0f8a27d81fcec1a" member_type: NON_VOTER last_known_addr { host: "127.6.117.198" port: 33655 } attrs { promote: true } }
I20240502 01:59:34.202504 10053 leader_election.cc:288] T b4409766ce994f698c31aede604a6927 P bdbecaeb8f214529b2012a9c003de2ff [CANDIDATE]: Term 2 election: Requested vote from peers 4b7f7ba2ac084e829c4304c16a94894f (127.6.117.194:46805), e77d1c0d41034d1eb5a036dc17c071f5 (127.6.117.195:43977)
I20240502 01:59:34.202759 10129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4409766ce994f698c31aede604a6927" candidate_uuid: "bdbecaeb8f214529b2012a9c003de2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: true dest_uuid: "4b7f7ba2ac084e829c4304c16a94894f"
I20240502 01:59:34.202868 10129 raft_consensus.cc:3037] T b4409766ce994f698c31aede604a6927 P 4b7f7ba2ac084e829c4304c16a94894f [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:59:34.202818 10200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4409766ce994f698c31aede604a6927" candidate_uuid: "bdbecaeb8f214529b2012a9c003de2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: true dest_uuid: "e77d1c0d41034d1eb5a036dc17c071f5"
I20240502 01:59:34.203003 10200 raft_consensus.cc:3032] T b4409766ce994f698c31aede604a6927 P e77d1c0d41034d1eb5a036dc17c071f5 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:59:34.203048 10200 raft_consensus.cc:729] T b4409766ce994f698c31aede604a6927 P e77d1c0d41034d1eb5a036dc17c071f5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e77d1c0d41034d1eb5a036dc17c071f5, State: Running, Role: LEADER
I20240502 01:59:34.203186 10200 consensus_queue.cc:260] T b4409766ce994f698c31aede604a6927 P e77d1c0d41034d1eb5a036dc17c071f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } attrs { replace: true } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "ae04d48c97204162b0f8a27d81fcec1a" member_type: NON_VOTER last_known_addr { host: "127.6.117.198" port: 33655 } attrs { promote: true } }
I20240502 01:59:34.203387 10200 raft_consensus.cc:3037] T b4409766ce994f698c31aede604a6927 P e77d1c0d41034d1eb5a036dc17c071f5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:59:34.203841 10129 raft_consensus.cc:2454] T b4409766ce994f698c31aede604a6927 P 4b7f7ba2ac084e829c4304c16a94894f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdbecaeb8f214529b2012a9c003de2ff in term 2.
I20240502 01:59:34.204133 10022 leader_election.cc:302] T b4409766ce994f698c31aede604a6927 P bdbecaeb8f214529b2012a9c003de2ff [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b7f7ba2ac084e829c4304c16a94894f, bdbecaeb8f214529b2012a9c003de2ff; no voters:
I20240502 01:59:34.204371 10679 raft_consensus.cc:2781] T b4409766ce994f698c31aede604a6927 P bdbecaeb8f214529b2012a9c003de2ff [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:59:34.204433 10200 raft_consensus.cc:2454] T b4409766ce994f698c31aede604a6927 P e77d1c0d41034d1eb5a036dc17c071f5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdbecaeb8f214529b2012a9c003de2ff in term 2.
I20240502 01:59:34.204476 10679 raft_consensus.cc:686] T b4409766ce994f698c31aede604a6927 P bdbecaeb8f214529b2012a9c003de2ff [term 2 LEADER]: Becoming Leader. State: Replica: bdbecaeb8f214529b2012a9c003de2ff, State: Running, Role: LEADER
I20240502 01:59:34.204679 10679 consensus_queue.cc:237] T b4409766ce994f698c31aede604a6927 P bdbecaeb8f214529b2012a9c003de2ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } attrs { replace: true } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "ae04d48c97204162b0f8a27d81fcec1a" member_type: NON_VOTER last_known_addr { host: "127.6.117.198" port: 33655 } attrs { promote: true } }
I20240502 01:59:34.206393 9952 catalog_manager.cc:5484] T b4409766ce994f698c31aede604a6927 P bdbecaeb8f214529b2012a9c003de2ff reported cstate change: term changed from 1 to 2, leader changed from e77d1c0d41034d1eb5a036dc17c071f5 (127.6.117.195) to bdbecaeb8f214529b2012a9c003de2ff (127.6.117.193). New cstate: current_term: 2 leader_uuid: "bdbecaeb8f214529b2012a9c003de2ff" committed_config { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae04d48c97204162b0f8a27d81fcec1a" member_type: NON_VOTER last_known_addr { host: "127.6.117.198" port: 33655 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:59:34.211566 10053 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b4409766ce994f698c31aede604a6927"
dest_uuid: "bdbecaeb8f214529b2012a9c003de2ff"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:44390
I20240502 01:59:34.211700 10053 raft_consensus.cc:595] T b4409766ce994f698c31aede604a6927 P bdbecaeb8f214529b2012a9c003de2ff [term 2 LEADER]: Received request to transfer leadership
I20240502 01:59:34.215687 10053 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b4409766ce994f698c31aede604a6927"
dest_uuid: "bdbecaeb8f214529b2012a9c003de2ff"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:44390
I20240502 01:59:34.215829 10053 raft_consensus.cc:595] T b4409766ce994f698c31aede604a6927 P bdbecaeb8f214529b2012a9c003de2ff [term 2 LEADER]: Received request to transfer leadership
I20240502 01:59:34.216517 10053 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b4409766ce994f698c31aede604a6927"
dest_uuid: "bdbecaeb8f214529b2012a9c003de2ff"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:44390
I20240502 01:59:34.216630 10053 raft_consensus.cc:595] T b4409766ce994f698c31aede604a6927 P bdbecaeb8f214529b2012a9c003de2ff [term 2 LEADER]: Received request to transfer leadership
W20240502 01:59:34.216902 10003 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for b4409766ce994f698c31aede604a6927 already in progress
W20240502 01:59:34.216953 10003 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for b4409766ce994f698c31aede604a6927 already in progress
I20240502 01:59:34.459676 10200 raft_consensus.cc:1261] T e7a8fc5afcab4b1db4ebf245b66df4be P e77d1c0d41034d1eb5a036dc17c071f5 [term 2 FOLLOWER]: Refusing update from remote peer 4b7f7ba2ac084e829c4304c16a94894f: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240502 01:59:34.460100 10393 consensus_queue.cc:1035] T e7a8fc5afcab4b1db4ebf245b66df4be P 4b7f7ba2ac084e829c4304c16a94894f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 01:59:34.461912 10677 raft_consensus.cc:979] T e7a8fc5afcab4b1db4ebf245b66df4be P 4b7f7ba2ac084e829c4304c16a94894f: : Instructing follower e77d1c0d41034d1eb5a036dc17c071f5 to start an election
I20240502 01:59:34.462030 10677 raft_consensus.cc:1067] T e7a8fc5afcab4b1db4ebf245b66df4be P 4b7f7ba2ac084e829c4304c16a94894f [term 2 LEADER]: Signalling peer e77d1c0d41034d1eb5a036dc17c071f5 to start an election
I20240502 01:59:34.462597 10202 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e7a8fc5afcab4b1db4ebf245b66df4be"
dest_uuid: "e77d1c0d41034d1eb5a036dc17c071f5"
from {username='slave'} at 127.0.0.1:48076
I20240502 01:59:34.462926 10202 raft_consensus.cc:482] T e7a8fc5afcab4b1db4ebf245b66df4be P e77d1c0d41034d1eb5a036dc17c071f5 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:59:34.463008 10202 raft_consensus.cc:3037] T e7a8fc5afcab4b1db4ebf245b66df4be P e77d1c0d41034d1eb5a036dc17c071f5 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:59:34.463173 10054 raft_consensus.cc:1261] T e7a8fc5afcab4b1db4ebf245b66df4be P bdbecaeb8f214529b2012a9c003de2ff [term 2 FOLLOWER]: Refusing update from remote peer 4b7f7ba2ac084e829c4304c16a94894f: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240502 01:59:34.463508 10676 consensus_queue.cc:1035] T e7a8fc5afcab4b1db4ebf245b66df4be P 4b7f7ba2ac084e829c4304c16a94894f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 01:59:34.464380 10202 raft_consensus.cc:504] T e7a8fc5afcab4b1db4ebf245b66df4be P e77d1c0d41034d1eb5a036dc17c071f5 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } attrs { replace: true } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "ae04d48c97204162b0f8a27d81fcec1a" member_type: NON_VOTER last_known_addr { host: "127.6.117.198" port: 33655 } attrs { promote: true } }
I20240502 01:59:34.464771 10202 leader_election.cc:288] T e7a8fc5afcab4b1db4ebf245b66df4be P e77d1c0d41034d1eb5a036dc17c071f5 [CANDIDATE]: Term 3 election: Requested vote from peers bdbecaeb8f214529b2012a9c003de2ff (127.6.117.193:34597), 4b7f7ba2ac084e829c4304c16a94894f (127.6.117.194:46805)
I20240502 01:59:34.465039 10131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7a8fc5afcab4b1db4ebf245b66df4be" candidate_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" candidate_term: 3 candidate_status { last_received { term: 2 index: 27 } } ignore_live_leader: true dest_uuid: "4b7f7ba2ac084e829c4304c16a94894f"
I20240502 01:59:34.465140 10131 raft_consensus.cc:3032] T e7a8fc5afcab4b1db4ebf245b66df4be P 4b7f7ba2ac084e829c4304c16a94894f [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:59:34.465178 10131 raft_consensus.cc:729] T e7a8fc5afcab4b1db4ebf245b66df4be P 4b7f7ba2ac084e829c4304c16a94894f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4b7f7ba2ac084e829c4304c16a94894f, State: Running, Role: LEADER
I20240502 01:59:34.465301 10131 consensus_queue.cc:260] T e7a8fc5afcab4b1db4ebf245b66df4be P 4b7f7ba2ac084e829c4304c16a94894f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 27, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } attrs { replace: true } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "ae04d48c97204162b0f8a27d81fcec1a" member_type: NON_VOTER last_known_addr { host: "127.6.117.198" port: 33655 } attrs { promote: true } }
I20240502 01:59:34.465427 10131 raft_consensus.cc:3037] T e7a8fc5afcab4b1db4ebf245b66df4be P 4b7f7ba2ac084e829c4304c16a94894f [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:59:34.466476 10131 raft_consensus.cc:2454] T e7a8fc5afcab4b1db4ebf245b66df4be P 4b7f7ba2ac084e829c4304c16a94894f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e77d1c0d41034d1eb5a036dc17c071f5 in term 3.
I20240502 01:59:34.466651 10053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7a8fc5afcab4b1db4ebf245b66df4be" candidate_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" candidate_term: 3 candidate_status { last_received { term: 2 index: 27 } } ignore_live_leader: true dest_uuid: "bdbecaeb8f214529b2012a9c003de2ff"
I20240502 01:59:34.466743 10053 raft_consensus.cc:3037] T e7a8fc5afcab4b1db4ebf245b66df4be P bdbecaeb8f214529b2012a9c003de2ff [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:59:34.467649 10053 raft_consensus.cc:2454] T e7a8fc5afcab4b1db4ebf245b66df4be P bdbecaeb8f214529b2012a9c003de2ff [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e77d1c0d41034d1eb5a036dc17c071f5 in term 3.
I20240502 01:59:34.467892 10174 leader_election.cc:302] T e7a8fc5afcab4b1db4ebf245b66df4be P e77d1c0d41034d1eb5a036dc17c071f5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bdbecaeb8f214529b2012a9c003de2ff, e77d1c0d41034d1eb5a036dc17c071f5; no voters:
I20240502 01:59:34.468061 10681 raft_consensus.cc:2781] T e7a8fc5afcab4b1db4ebf245b66df4be P e77d1c0d41034d1eb5a036dc17c071f5 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:59:34.468137 10681 raft_consensus.cc:686] T e7a8fc5afcab4b1db4ebf245b66df4be P e77d1c0d41034d1eb5a036dc17c071f5 [term 3 LEADER]: Becoming Leader. State: Replica: e77d1c0d41034d1eb5a036dc17c071f5, State: Running, Role: LEADER
I20240502 01:59:34.468250 10681 consensus_queue.cc:237] T e7a8fc5afcab4b1db4ebf245b66df4be P e77d1c0d41034d1eb5a036dc17c071f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 27, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } attrs { replace: true } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "ae04d48c97204162b0f8a27d81fcec1a" member_type: NON_VOTER last_known_addr { host: "127.6.117.198" port: 33655 } attrs { promote: true } }
I20240502 01:59:34.469969 9952 catalog_manager.cc:5484] T e7a8fc5afcab4b1db4ebf245b66df4be P e77d1c0d41034d1eb5a036dc17c071f5 reported cstate change: term changed from 2 to 3, leader changed from 4b7f7ba2ac084e829c4304c16a94894f (127.6.117.194) to e77d1c0d41034d1eb5a036dc17c071f5 (127.6.117.195). New cstate: current_term: 3 leader_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae04d48c97204162b0f8a27d81fcec1a" member_type: NON_VOTER last_known_addr { host: "127.6.117.198" port: 33655 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:59:34.642905 10131 raft_consensus.cc:1261] T b4409766ce994f698c31aede604a6927 P 4b7f7ba2ac084e829c4304c16a94894f [term 2 FOLLOWER]: Refusing update from remote peer bdbecaeb8f214529b2012a9c003de2ff: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20240502 01:59:34.643308 10691 consensus_queue.cc:1035] T b4409766ce994f698c31aede604a6927 P bdbecaeb8f214529b2012a9c003de2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 01:59:34.644856 10679 raft_consensus.cc:979] T b4409766ce994f698c31aede604a6927 P bdbecaeb8f214529b2012a9c003de2ff: : Instructing follower 4b7f7ba2ac084e829c4304c16a94894f to start an election
I20240502 01:59:34.644950 10679 raft_consensus.cc:1067] T b4409766ce994f698c31aede604a6927 P bdbecaeb8f214529b2012a9c003de2ff [term 2 LEADER]: Signalling peer 4b7f7ba2ac084e829c4304c16a94894f to start an election
I20240502 01:59:34.645370 10129 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b4409766ce994f698c31aede604a6927"
dest_uuid: "4b7f7ba2ac084e829c4304c16a94894f"
from {username='slave'} at 127.0.0.1:52602
I20240502 01:59:34.645788 10129 raft_consensus.cc:482] T b4409766ce994f698c31aede604a6927 P 4b7f7ba2ac084e829c4304c16a94894f [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:59:34.645864 10129 raft_consensus.cc:3037] T b4409766ce994f698c31aede604a6927 P 4b7f7ba2ac084e829c4304c16a94894f [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:59:34.646428 10202 raft_consensus.cc:1261] T b4409766ce994f698c31aede604a6927 P e77d1c0d41034d1eb5a036dc17c071f5 [term 2 FOLLOWER]: Refusing update from remote peer bdbecaeb8f214529b2012a9c003de2ff: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
W20240502 01:59:34.646482 10021 consensus_peers.cc:485] T b4409766ce994f698c31aede604a6927 P bdbecaeb8f214529b2012a9c003de2ff -> Peer ae04d48c97204162b0f8a27d81fcec1a (127.6.117.198:33655): Couldn't send request to peer ae04d48c97204162b0f8a27d81fcec1a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:34.646988 10129 raft_consensus.cc:504] T b4409766ce994f698c31aede604a6927 P 4b7f7ba2ac084e829c4304c16a94894f [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } attrs { replace: true } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "ae04d48c97204162b0f8a27d81fcec1a" member_type: NON_VOTER last_known_addr { host: "127.6.117.198" port: 33655 } attrs { promote: true } }
I20240502 01:59:34.647351 10129 leader_election.cc:288] T b4409766ce994f698c31aede604a6927 P 4b7f7ba2ac084e829c4304c16a94894f [CANDIDATE]: Term 3 election: Requested vote from peers bdbecaeb8f214529b2012a9c003de2ff (127.6.117.193:34597), e77d1c0d41034d1eb5a036dc17c071f5 (127.6.117.195:43977)
I20240502 01:59:34.647770 10053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4409766ce994f698c31aede604a6927" candidate_uuid: "4b7f7ba2ac084e829c4304c16a94894f" candidate_term: 3 candidate_status { last_received { term: 2 index: 26 } } ignore_live_leader: true dest_uuid: "bdbecaeb8f214529b2012a9c003de2ff"
I20240502 01:59:34.647783 10679 consensus_queue.cc:1035] T b4409766ce994f698c31aede604a6927 P bdbecaeb8f214529b2012a9c003de2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 01:59:34.648052 10053 raft_consensus.cc:3032] T b4409766ce994f698c31aede604a6927 P bdbecaeb8f214529b2012a9c003de2ff [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:59:34.648097 10053 raft_consensus.cc:729] T b4409766ce994f698c31aede604a6927 P bdbecaeb8f214529b2012a9c003de2ff [term 2 LEADER]: Becoming Follower/Learner. State: Replica: bdbecaeb8f214529b2012a9c003de2ff, State: Running, Role: LEADER
I20240502 01:59:34.648211 10053 consensus_queue.cc:260] T b4409766ce994f698c31aede604a6927 P bdbecaeb8f214529b2012a9c003de2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 2.26, Last appended by leader: 26, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } attrs { replace: true } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "ae04d48c97204162b0f8a27d81fcec1a" member_type: NON_VOTER last_known_addr { host: "127.6.117.198" port: 33655 } attrs { promote: true } }
I20240502 01:59:34.648399 10053 raft_consensus.cc:3037] T b4409766ce994f698c31aede604a6927 P bdbecaeb8f214529b2012a9c003de2ff [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:59:34.648455 10202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4409766ce994f698c31aede604a6927" candidate_uuid: "4b7f7ba2ac084e829c4304c16a94894f" candidate_term: 3 candidate_status { last_received { term: 2 index: 26 } } ignore_live_leader: true dest_uuid: "e77d1c0d41034d1eb5a036dc17c071f5"
I20240502 01:59:34.648538 10202 raft_consensus.cc:3037] T b4409766ce994f698c31aede604a6927 P e77d1c0d41034d1eb5a036dc17c071f5 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:59:34.649394 10053 raft_consensus.cc:2454] T b4409766ce994f698c31aede604a6927 P bdbecaeb8f214529b2012a9c003de2ff [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b7f7ba2ac084e829c4304c16a94894f in term 3.
I20240502 01:59:34.649829 10099 leader_election.cc:302] T b4409766ce994f698c31aede604a6927 P 4b7f7ba2ac084e829c4304c16a94894f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b7f7ba2ac084e829c4304c16a94894f, bdbecaeb8f214529b2012a9c003de2ff; no voters:
I20240502 01:59:34.650007 10677 raft_consensus.cc:2781] T b4409766ce994f698c31aede604a6927 P 4b7f7ba2ac084e829c4304c16a94894f [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:59:34.650117 10677 raft_consensus.cc:686] T b4409766ce994f698c31aede604a6927 P 4b7f7ba2ac084e829c4304c16a94894f [term 3 LEADER]: Becoming Leader. State: Replica: 4b7f7ba2ac084e829c4304c16a94894f, State: Running, Role: LEADER
I20240502 01:59:34.649394 10202 raft_consensus.cc:2454] T b4409766ce994f698c31aede604a6927 P e77d1c0d41034d1eb5a036dc17c071f5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b7f7ba2ac084e829c4304c16a94894f in term 3.
I20240502 01:59:34.650275 10677 consensus_queue.cc:237] T b4409766ce994f698c31aede604a6927 P 4b7f7ba2ac084e829c4304c16a94894f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 2.26, Last appended by leader: 26, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } attrs { replace: true } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "ae04d48c97204162b0f8a27d81fcec1a" member_type: NON_VOTER last_known_addr { host: "127.6.117.198" port: 33655 } attrs { promote: true } }
I20240502 01:59:34.650540 10200 raft_consensus.cc:1226] T b4409766ce994f698c31aede604a6927 P e77d1c0d41034d1eb5a036dc17c071f5 [term 3 FOLLOWER]: Rejecting Update request from peer bdbecaeb8f214529b2012a9c003de2ff for earlier term 2. Current term is 3. Ops: [2.26-2.26]
I20240502 01:59:34.651928 9953 catalog_manager.cc:5484] T b4409766ce994f698c31aede604a6927 P 4b7f7ba2ac084e829c4304c16a94894f reported cstate change: term changed from 2 to 3, leader changed from bdbecaeb8f214529b2012a9c003de2ff (127.6.117.193) to 4b7f7ba2ac084e829c4304c16a94894f (127.6.117.194). New cstate: current_term: 3 leader_uuid: "4b7f7ba2ac084e829c4304c16a94894f" committed_config { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae04d48c97204162b0f8a27d81fcec1a" member_type: NON_VOTER last_known_addr { host: "127.6.117.198" port: 33655 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 01:59:35.032142 10172 consensus_peers.cc:485] T e7a8fc5afcab4b1db4ebf245b66df4be P e77d1c0d41034d1eb5a036dc17c071f5 -> Peer ae04d48c97204162b0f8a27d81fcec1a (127.6.117.198:33655): Couldn't send request to peer ae04d48c97204162b0f8a27d81fcec1a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:35.032164 10129 raft_consensus.cc:1261] T e7a8fc5afcab4b1db4ebf245b66df4be P 4b7f7ba2ac084e829c4304c16a94894f [term 3 FOLLOWER]: Refusing update from remote peer e77d1c0d41034d1eb5a036dc17c071f5: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20240502 01:59:35.032626 10700 consensus_queue.cc:1035] T e7a8fc5afcab4b1db4ebf245b66df4be P e77d1c0d41034d1eb5a036dc17c071f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 01:59:35.035200 10054 raft_consensus.cc:1261] T e7a8fc5afcab4b1db4ebf245b66df4be P bdbecaeb8f214529b2012a9c003de2ff [term 3 FOLLOWER]: Refusing update from remote peer e77d1c0d41034d1eb5a036dc17c071f5: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20240502 01:59:35.035693 10702 consensus_queue.cc:1035] T e7a8fc5afcab4b1db4ebf245b66df4be P e77d1c0d41034d1eb5a036dc17c071f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
W20240502 01:59:35.124222 10097 consensus_peers.cc:485] T b4409766ce994f698c31aede604a6927 P 4b7f7ba2ac084e829c4304c16a94894f -> Peer ae04d48c97204162b0f8a27d81fcec1a (127.6.117.198:33655): Couldn't send request to peer ae04d48c97204162b0f8a27d81fcec1a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:35.148061 10200 raft_consensus.cc:1261] T b4409766ce994f698c31aede604a6927 P e77d1c0d41034d1eb5a036dc17c071f5 [term 3 FOLLOWER]: Refusing update from remote peer 4b7f7ba2ac084e829c4304c16a94894f: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 3 index: 27. (index mismatch)
I20240502 01:59:35.148443 10677 consensus_queue.cc:1035] T b4409766ce994f698c31aede604a6927 P 4b7f7ba2ac084e829c4304c16a94894f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 01:59:35.152165 10054 raft_consensus.cc:1261] T b4409766ce994f698c31aede604a6927 P bdbecaeb8f214529b2012a9c003de2ff [term 3 FOLLOWER]: Refusing update from remote peer 4b7f7ba2ac084e829c4304c16a94894f: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 3 index: 27. (index mismatch)
I20240502 01:59:35.152618 10704 consensus_queue.cc:1035] T b4409766ce994f698c31aede604a6927 P 4b7f7ba2ac084e829c4304c16a94894f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 01:59:35.321038 10642 tablet_copy_client.cc:523] T b4409766ce994f698c31aede604a6927 P ae04d48c97204162b0f8a27d81fcec1a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:59:35.321689 10643 tablet_copy_client.cc:523] T 73ea0bfb40ad43a68a6289ff1b35d005 P d6ff17402670432da6eeb5da2af7b4b6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:59:35.322712 10642 tablet_bootstrap.cc:492] T b4409766ce994f698c31aede604a6927 P ae04d48c97204162b0f8a27d81fcec1a: Bootstrap starting.
I20240502 01:59:35.323030 10643 tablet_bootstrap.cc:492] T 73ea0bfb40ad43a68a6289ff1b35d005 P d6ff17402670432da6eeb5da2af7b4b6: Bootstrap starting.
I20240502 01:59:35.344667 10643 tablet_bootstrap.cc:492] T 73ea0bfb40ad43a68a6289ff1b35d005 P d6ff17402670432da6eeb5da2af7b4b6: Bootstrap replayed 1/1 log segments. Stats: ops{read=25 overwritten=0 applied=25 ignored=0} inserts{seen=102 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:59:35.345033 10643 tablet_bootstrap.cc:492] T 73ea0bfb40ad43a68a6289ff1b35d005 P d6ff17402670432da6eeb5da2af7b4b6: Bootstrap complete.
I20240502 01:59:35.345232 10643 ts_tablet_manager.cc:1367] T 73ea0bfb40ad43a68a6289ff1b35d005 P d6ff17402670432da6eeb5da2af7b4b6: Time spent bootstrapping tablet: real 0.022s user 0.022s sys 0.000s
I20240502 01:59:35.345734 10643 raft_consensus.cc:348] T 73ea0bfb40ad43a68a6289ff1b35d005 P d6ff17402670432da6eeb5da2af7b4b6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "d6ff17402670432da6eeb5da2af7b4b6" member_type: NON_VOTER last_known_addr { host: "127.6.117.196" port: 34337 } attrs { promote: true } }
I20240502 01:59:35.345852 10643 raft_consensus.cc:729] T 73ea0bfb40ad43a68a6289ff1b35d005 P d6ff17402670432da6eeb5da2af7b4b6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d6ff17402670432da6eeb5da2af7b4b6, State: Initialized, Role: LEARNER
I20240502 01:59:35.345971 10643 consensus_queue.cc:260] T 73ea0bfb40ad43a68a6289ff1b35d005 P d6ff17402670432da6eeb5da2af7b4b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "d6ff17402670432da6eeb5da2af7b4b6" member_type: NON_VOTER last_known_addr { host: "127.6.117.196" port: 34337 } attrs { promote: true } }
I20240502 01:59:35.346133 10642 tablet_bootstrap.cc:492] T b4409766ce994f698c31aede604a6927 P ae04d48c97204162b0f8a27d81fcec1a: Bootstrap replayed 1/1 log segments. Stats: ops{read=25 overwritten=0 applied=25 ignored=0} inserts{seen=92 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:59:35.346365 10643 ts_tablet_manager.cc:1398] T 73ea0bfb40ad43a68a6289ff1b35d005 P d6ff17402670432da6eeb5da2af7b4b6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:35.346822 10642 tablet_bootstrap.cc:492] T b4409766ce994f698c31aede604a6927 P ae04d48c97204162b0f8a27d81fcec1a: Bootstrap complete.
I20240502 01:59:35.346897 10143 tablet_copy_service.cc:342] P 4b7f7ba2ac084e829c4304c16a94894f: Request end of tablet copy session d6ff17402670432da6eeb5da2af7b4b6-73ea0bfb40ad43a68a6289ff1b35d005 received from {username='slave'} at 127.0.0.1:52658
I20240502 01:59:35.347553 10143 tablet_copy_service.cc:433] P 4b7f7ba2ac084e829c4304c16a94894f: ending tablet copy session d6ff17402670432da6eeb5da2af7b4b6-73ea0bfb40ad43a68a6289ff1b35d005 on tablet 73ea0bfb40ad43a68a6289ff1b35d005 with peer d6ff17402670432da6eeb5da2af7b4b6
I20240502 01:59:35.347690 10642 ts_tablet_manager.cc:1367] T b4409766ce994f698c31aede604a6927 P ae04d48c97204162b0f8a27d81fcec1a: Time spent bootstrapping tablet: real 0.025s user 0.024s sys 0.000s
I20240502 01:59:35.348155 10642 raft_consensus.cc:348] T b4409766ce994f698c31aede604a6927 P ae04d48c97204162b0f8a27d81fcec1a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } attrs { replace: true } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "ae04d48c97204162b0f8a27d81fcec1a" member_type: NON_VOTER last_known_addr { host: "127.6.117.198" port: 33655 } attrs { promote: true } }
I20240502 01:59:35.348332 10642 raft_consensus.cc:729] T b4409766ce994f698c31aede604a6927 P ae04d48c97204162b0f8a27d81fcec1a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ae04d48c97204162b0f8a27d81fcec1a, State: Initialized, Role: LEARNER
I20240502 01:59:35.348455 10642 consensus_queue.cc:260] T b4409766ce994f698c31aede604a6927 P ae04d48c97204162b0f8a27d81fcec1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } attrs { replace: true } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "ae04d48c97204162b0f8a27d81fcec1a" member_type: NON_VOTER last_known_addr { host: "127.6.117.198" port: 33655 } attrs { promote: true } }
I20240502 01:59:35.348874 10642 ts_tablet_manager.cc:1398] T b4409766ce994f698c31aede604a6927 P ae04d48c97204162b0f8a27d81fcec1a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:35.349284 10218 tablet_copy_service.cc:342] P e77d1c0d41034d1eb5a036dc17c071f5: Request end of tablet copy session ae04d48c97204162b0f8a27d81fcec1a-b4409766ce994f698c31aede604a6927 received from {username='slave'} at 127.0.0.1:48102
I20240502 01:59:35.349366 10218 tablet_copy_service.cc:433] P e77d1c0d41034d1eb5a036dc17c071f5: ending tablet copy session ae04d48c97204162b0f8a27d81fcec1a-b4409766ce994f698c31aede604a6927 on tablet b4409766ce994f698c31aede604a6927 with peer ae04d48c97204162b0f8a27d81fcec1a
I20240502 01:59:35.439005 10654 tablet_copy_client.cc:523] T 580db11a61cb46f48d5f7507cfbdbb52 P d52d70d7578f4995b03ac6faba3ec739: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:59:35.440759 10654 tablet_bootstrap.cc:492] T 580db11a61cb46f48d5f7507cfbdbb52 P d52d70d7578f4995b03ac6faba3ec739: Bootstrap starting.
I20240502 01:59:35.463551 10654 tablet_bootstrap.cc:492] T 580db11a61cb46f48d5f7507cfbdbb52 P d52d70d7578f4995b03ac6faba3ec739: Bootstrap replayed 1/1 log segments. Stats: ops{read=26 overwritten=0 applied=26 ignored=0} inserts{seen=105 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:59:35.463867 10654 tablet_bootstrap.cc:492] T 580db11a61cb46f48d5f7507cfbdbb52 P d52d70d7578f4995b03ac6faba3ec739: Bootstrap complete.
I20240502 01:59:35.464120 10654 ts_tablet_manager.cc:1367] T 580db11a61cb46f48d5f7507cfbdbb52 P d52d70d7578f4995b03ac6faba3ec739: Time spent bootstrapping tablet: real 0.023s user 0.019s sys 0.004s
I20240502 01:59:35.464633 10654 raft_consensus.cc:348] T 580db11a61cb46f48d5f7507cfbdbb52 P d52d70d7578f4995b03ac6faba3ec739 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } attrs { replace: true } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "d52d70d7578f4995b03ac6faba3ec739" member_type: NON_VOTER last_known_addr { host: "127.6.117.197" port: 44467 } attrs { promote: true } }
I20240502 01:59:35.464748 10654 raft_consensus.cc:729] T 580db11a61cb46f48d5f7507cfbdbb52 P d52d70d7578f4995b03ac6faba3ec739 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d52d70d7578f4995b03ac6faba3ec739, State: Initialized, Role: LEARNER
I20240502 01:59:35.464841 10654 consensus_queue.cc:260] T 580db11a61cb46f48d5f7507cfbdbb52 P d52d70d7578f4995b03ac6faba3ec739 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } attrs { replace: true } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "d52d70d7578f4995b03ac6faba3ec739" member_type: NON_VOTER last_known_addr { host: "127.6.117.197" port: 44467 } attrs { promote: true } }
I20240502 01:59:35.465524 10654 ts_tablet_manager.cc:1398] T 580db11a61cb46f48d5f7507cfbdbb52 P d52d70d7578f4995b03ac6faba3ec739: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:59:35.466449 10067 tablet_copy_service.cc:342] P bdbecaeb8f214529b2012a9c003de2ff: Request end of tablet copy session d52d70d7578f4995b03ac6faba3ec739-580db11a61cb46f48d5f7507cfbdbb52 received from {username='slave'} at 127.0.0.1:44448
I20240502 01:59:35.466521 10067 tablet_copy_service.cc:433] P bdbecaeb8f214529b2012a9c003de2ff: ending tablet copy session d52d70d7578f4995b03ac6faba3ec739-580db11a61cb46f48d5f7507cfbdbb52 on tablet 580db11a61cb46f48d5f7507cfbdbb52 with peer d52d70d7578f4995b03ac6faba3ec739
I20240502 01:59:35.515487 10661 tablet_copy_client.cc:523] T e7a8fc5afcab4b1db4ebf245b66df4be P ae04d48c97204162b0f8a27d81fcec1a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:59:35.517194 10661 tablet_bootstrap.cc:492] T e7a8fc5afcab4b1db4ebf245b66df4be P ae04d48c97204162b0f8a27d81fcec1a: Bootstrap starting.
I20240502 01:59:35.540359 10661 tablet_bootstrap.cc:492] T e7a8fc5afcab4b1db4ebf245b66df4be P ae04d48c97204162b0f8a27d81fcec1a: Bootstrap replayed 1/1 log segments. Stats: ops{read=26 overwritten=0 applied=26 ignored=0} inserts{seen=97 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:59:35.540653 10661 tablet_bootstrap.cc:492] T e7a8fc5afcab4b1db4ebf245b66df4be P ae04d48c97204162b0f8a27d81fcec1a: Bootstrap complete.
I20240502 01:59:35.540850 10661 ts_tablet_manager.cc:1367] T e7a8fc5afcab4b1db4ebf245b66df4be P ae04d48c97204162b0f8a27d81fcec1a: Time spent bootstrapping tablet: real 0.024s user 0.022s sys 0.001s
I20240502 01:59:35.541360 10661 raft_consensus.cc:348] T e7a8fc5afcab4b1db4ebf245b66df4be P ae04d48c97204162b0f8a27d81fcec1a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } attrs { replace: true } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "ae04d48c97204162b0f8a27d81fcec1a" member_type: NON_VOTER last_known_addr { host: "127.6.117.198" port: 33655 } attrs { promote: true } }
I20240502 01:59:35.541486 10661 raft_consensus.cc:729] T e7a8fc5afcab4b1db4ebf245b66df4be P ae04d48c97204162b0f8a27d81fcec1a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ae04d48c97204162b0f8a27d81fcec1a, State: Initialized, Role: LEARNER
I20240502 01:59:35.541579 10661 consensus_queue.cc:260] T e7a8fc5afcab4b1db4ebf245b66df4be P ae04d48c97204162b0f8a27d81fcec1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } attrs { replace: true } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "ae04d48c97204162b0f8a27d81fcec1a" member_type: NON_VOTER last_known_addr { host: "127.6.117.198" port: 33655 } attrs { promote: true } }
I20240502 01:59:35.541903 10661 ts_tablet_manager.cc:1398] T e7a8fc5afcab4b1db4ebf245b66df4be P ae04d48c97204162b0f8a27d81fcec1a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:59:35.542361 10067 tablet_copy_service.cc:342] P bdbecaeb8f214529b2012a9c003de2ff: Request end of tablet copy session ae04d48c97204162b0f8a27d81fcec1a-e7a8fc5afcab4b1db4ebf245b66df4be received from {username='slave'} at 127.0.0.1:44462
I20240502 01:59:35.542441 10067 tablet_copy_service.cc:433] P bdbecaeb8f214529b2012a9c003de2ff: ending tablet copy session ae04d48c97204162b0f8a27d81fcec1a-e7a8fc5afcab4b1db4ebf245b66df4be on tablet e7a8fc5afcab4b1db4ebf245b66df4be with peer ae04d48c97204162b0f8a27d81fcec1a
I20240502 01:59:35.554957 10666 tablet_copy_client.cc:523] T 1851ea6467584b0ea32a1f68696803d0 P d6ff17402670432da6eeb5da2af7b4b6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:59:35.556797 10666 tablet_bootstrap.cc:492] T 1851ea6467584b0ea32a1f68696803d0 P d6ff17402670432da6eeb5da2af7b4b6: Bootstrap starting.
I20240502 01:59:35.560155 10592 raft_consensus.cc:3037] T b4409766ce994f698c31aede604a6927 P ae04d48c97204162b0f8a27d81fcec1a [term 1 LEARNER]: Advancing to term 3
I20240502 01:59:35.561203 10592 raft_consensus.cc:1261] T b4409766ce994f698c31aede604a6927 P ae04d48c97204162b0f8a27d81fcec1a [term 3 LEARNER]: Refusing update from remote peer 4b7f7ba2ac084e829c4304c16a94894f: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20240502 01:59:35.561785 10677 consensus_queue.cc:1037] T b4409766ce994f698c31aede604a6927 P 4b7f7ba2ac084e829c4304c16a94894f [LEADER]: Got LMP mismatch error from peer: Peer: permanent_uuid: "ae04d48c97204162b0f8a27d81fcec1a" member_type: NON_VOTER last_known_addr { host: "127.6.117.198" port: 33655 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 01:59:35.564045 10677 raft_consensus.cc:1050] T b4409766ce994f698c31aede604a6927 P 4b7f7ba2ac084e829c4304c16a94894f: attempting to promote NON_VOTER ae04d48c97204162b0f8a27d81fcec1a to VOTER
I20240502 01:59:35.564473 10677 consensus_queue.cc:237] T b4409766ce994f698c31aede604a6927 P 4b7f7ba2ac084e829c4304c16a94894f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 3.27, Last appended by leader: 26, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } attrs { replace: true } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "ae04d48c97204162b0f8a27d81fcec1a" member_type: VOTER last_known_addr { host: "127.6.117.198" port: 33655 } attrs { promote: false } }
I20240502 01:59:35.565783 10200 raft_consensus.cc:1261] T b4409766ce994f698c31aede604a6927 P e77d1c0d41034d1eb5a036dc17c071f5 [term 3 FOLLOWER]: Refusing update from remote peer 4b7f7ba2ac084e829c4304c16a94894f: Log matching property violated. Preceding OpId in replica: term: 3 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20240502 01:59:35.566100 10053 raft_consensus.cc:1261] T b4409766ce994f698c31aede604a6927 P bdbecaeb8f214529b2012a9c003de2ff [term 3 FOLLOWER]: Refusing update from remote peer 4b7f7ba2ac084e829c4304c16a94894f: Log matching property violated. Preceding OpId in replica: term: 3 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20240502 01:59:35.566304 10704 consensus_queue.cc:1035] T b4409766ce994f698c31aede604a6927 P 4b7f7ba2ac084e829c4304c16a94894f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 01:59:35.566126 10592 raft_consensus.cc:1261] T b4409766ce994f698c31aede604a6927 P ae04d48c97204162b0f8a27d81fcec1a [term 3 LEARNER]: Refusing update from remote peer 4b7f7ba2ac084e829c4304c16a94894f: Log matching property violated. Preceding OpId in replica: term: 3 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20240502 01:59:35.567045 10704 consensus_queue.cc:1035] T b4409766ce994f698c31aede604a6927 P 4b7f7ba2ac084e829c4304c16a94894f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 01:59:35.568303 10704 consensus_queue.cc:1035] T b4409766ce994f698c31aede604a6927 P 4b7f7ba2ac084e829c4304c16a94894f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae04d48c97204162b0f8a27d81fcec1a" member_type: VOTER last_known_addr { host: "127.6.117.198" port: 33655 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 01:59:35.569751 10705 raft_consensus.cc:2932] T b4409766ce994f698c31aede604a6927 P 4b7f7ba2ac084e829c4304c16a94894f [term 3 LEADER]: Committing config change with OpId 3.28: config changed from index 25 to 28, ae04d48c97204162b0f8a27d81fcec1a (127.6.117.198) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } attrs { replace: true } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "ae04d48c97204162b0f8a27d81fcec1a" member_type: VOTER last_known_addr { host: "127.6.117.198" port: 33655 } attrs { promote: false } } }
I20240502 01:59:35.570150 10054 raft_consensus.cc:2932] T b4409766ce994f698c31aede604a6927 P bdbecaeb8f214529b2012a9c003de2ff [term 3 FOLLOWER]: Committing config change with OpId 3.28: config changed from index 25 to 28, ae04d48c97204162b0f8a27d81fcec1a (127.6.117.198) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } attrs { replace: true } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "ae04d48c97204162b0f8a27d81fcec1a" member_type: VOTER last_known_addr { host: "127.6.117.198" port: 33655 } attrs { promote: false } } }
I20240502 01:59:35.572283 9952 catalog_manager.cc:5484] T b4409766ce994f698c31aede604a6927 P 4b7f7ba2ac084e829c4304c16a94894f reported cstate change: config changed from index 25 to 28, ae04d48c97204162b0f8a27d81fcec1a (127.6.117.198) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "4b7f7ba2ac084e829c4304c16a94894f" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae04d48c97204162b0f8a27d81fcec1a" member_type: VOTER last_known_addr { host: "127.6.117.198" port: 33655 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:59:35.573026 10200 raft_consensus.cc:2932] T b4409766ce994f698c31aede604a6927 P e77d1c0d41034d1eb5a036dc17c071f5 [term 3 FOLLOWER]: Committing config change with OpId 3.28: config changed from index 25 to 28, ae04d48c97204162b0f8a27d81fcec1a (127.6.117.198) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } attrs { replace: true } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "ae04d48c97204162b0f8a27d81fcec1a" member_type: VOTER last_known_addr { host: "127.6.117.198" port: 33655 } attrs { promote: false } } }
I20240502 01:59:35.573385 10671 tablet_copy_client.cc:523] T 19ff9c216a094fde86ea9f9648bf08cb P d52d70d7578f4995b03ac6faba3ec739: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:59:35.574790 10671 tablet_bootstrap.cc:492] T 19ff9c216a094fde86ea9f9648bf08cb P d52d70d7578f4995b03ac6faba3ec739: Bootstrap starting.
I20240502 01:59:35.575214 10592 raft_consensus.cc:3037] T e7a8fc5afcab4b1db4ebf245b66df4be P ae04d48c97204162b0f8a27d81fcec1a [term 1 LEARNER]: Advancing to term 3
I20240502 01:59:35.576244 10592 raft_consensus.cc:1261] T e7a8fc5afcab4b1db4ebf245b66df4be P ae04d48c97204162b0f8a27d81fcec1a [term 3 LEARNER]: Refusing update from remote peer e77d1c0d41034d1eb5a036dc17c071f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240502 01:59:35.576699 10722 consensus_queue.cc:1037] T e7a8fc5afcab4b1db4ebf245b66df4be P e77d1c0d41034d1eb5a036dc17c071f5 [LEADER]: Got LMP mismatch error from peer: Peer: permanent_uuid: "ae04d48c97204162b0f8a27d81fcec1a" member_type: NON_VOTER last_known_addr { host: "127.6.117.198" port: 33655 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 01:59:35.579030 10722 raft_consensus.cc:1050] T e7a8fc5afcab4b1db4ebf245b66df4be P e77d1c0d41034d1eb5a036dc17c071f5: attempting to promote NON_VOTER ae04d48c97204162b0f8a27d81fcec1a to VOTER
I20240502 01:59:35.579458 10722 consensus_queue.cc:237] T e7a8fc5afcab4b1db4ebf245b66df4be P e77d1c0d41034d1eb5a036dc17c071f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 3.28, Last appended by leader: 27, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } attrs { replace: true } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "ae04d48c97204162b0f8a27d81fcec1a" member_type: VOTER last_known_addr { host: "127.6.117.198" port: 33655 } attrs { promote: false } }
I20240502 01:59:35.580742 10131 raft_consensus.cc:1261] T e7a8fc5afcab4b1db4ebf245b66df4be P 4b7f7ba2ac084e829c4304c16a94894f [term 3 FOLLOWER]: Refusing update from remote peer e77d1c0d41034d1eb5a036dc17c071f5: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240502 01:59:35.581110 10702 consensus_queue.cc:1035] T e7a8fc5afcab4b1db4ebf245b66df4be P e77d1c0d41034d1eb5a036dc17c071f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 01:59:35.581598 10592 raft_consensus.cc:1261] T e7a8fc5afcab4b1db4ebf245b66df4be P ae04d48c97204162b0f8a27d81fcec1a [term 3 LEARNER]: Refusing update from remote peer e77d1c0d41034d1eb5a036dc17c071f5: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240502 01:59:35.581939 10702 consensus_queue.cc:1035] T e7a8fc5afcab4b1db4ebf245b66df4be P e77d1c0d41034d1eb5a036dc17c071f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae04d48c97204162b0f8a27d81fcec1a" member_type: VOTER last_known_addr { host: "127.6.117.198" port: 33655 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 01:59:35.582000 10054 raft_consensus.cc:1261] T e7a8fc5afcab4b1db4ebf245b66df4be P bdbecaeb8f214529b2012a9c003de2ff [term 3 FOLLOWER]: Refusing update from remote peer e77d1c0d41034d1eb5a036dc17c071f5: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240502 01:59:35.582307 10702 consensus_queue.cc:1035] T e7a8fc5afcab4b1db4ebf245b66df4be P e77d1c0d41034d1eb5a036dc17c071f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 01:59:35.584085 10592 raft_consensus.cc:2932] T b4409766ce994f698c31aede604a6927 P ae04d48c97204162b0f8a27d81fcec1a [term 3 FOLLOWER]: Committing config change with OpId 3.28: config changed from index 25 to 28, ae04d48c97204162b0f8a27d81fcec1a (127.6.117.198) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } attrs { replace: true } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "ae04d48c97204162b0f8a27d81fcec1a" member_type: VOTER last_known_addr { host: "127.6.117.198" port: 33655 } attrs { promote: false } } }
I20240502 01:59:35.584306 10702 raft_consensus.cc:2932] T e7a8fc5afcab4b1db4ebf245b66df4be P e77d1c0d41034d1eb5a036dc17c071f5 [term 3 LEADER]: Committing config change with OpId 3.29: config changed from index 26 to 29, ae04d48c97204162b0f8a27d81fcec1a (127.6.117.198) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } attrs { replace: true } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "ae04d48c97204162b0f8a27d81fcec1a" member_type: VOTER last_known_addr { host: "127.6.117.198" port: 33655 } attrs { promote: false } } }
I20240502 01:59:35.584434 10129 consensus_queue.cc:237] T b4409766ce994f698c31aede604a6927 P 4b7f7ba2ac084e829c4304c16a94894f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 3.28, Last appended by leader: 26, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "ae04d48c97204162b0f8a27d81fcec1a" member_type: VOTER last_known_addr { host: "127.6.117.198" port: 33655 } attrs { promote: false } }
I20240502 01:59:35.585641 10200 raft_consensus.cc:1261] T b4409766ce994f698c31aede604a6927 P e77d1c0d41034d1eb5a036dc17c071f5 [term 3 FOLLOWER]: Refusing update from remote peer 4b7f7ba2ac084e829c4304c16a94894f: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240502 01:59:35.586077 10704 consensus_queue.cc:1035] T b4409766ce994f698c31aede604a6927 P 4b7f7ba2ac084e829c4304c16a94894f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 01:59:35.586261 10591 raft_consensus.cc:2932] T e7a8fc5afcab4b1db4ebf245b66df4be P ae04d48c97204162b0f8a27d81fcec1a [term 3 FOLLOWER]: Committing config change with OpId 3.29: config changed from index 26 to 29, ae04d48c97204162b0f8a27d81fcec1a (127.6.117.198) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } attrs { replace: true } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "ae04d48c97204162b0f8a27d81fcec1a" member_type: VOTER last_known_addr { host: "127.6.117.198" port: 33655 } attrs { promote: false } } }
I20240502 01:59:35.587656 10590 raft_consensus.cc:1261] T b4409766ce994f698c31aede604a6927 P ae04d48c97204162b0f8a27d81fcec1a [term 3 FOLLOWER]: Refusing update from remote peer 4b7f7ba2ac084e829c4304c16a94894f: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240502 01:59:35.587953 10704 consensus_queue.cc:1035] T b4409766ce994f698c31aede604a6927 P 4b7f7ba2ac084e829c4304c16a94894f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae04d48c97204162b0f8a27d81fcec1a" member_type: VOTER last_known_addr { host: "127.6.117.198" port: 33655 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 01:59:35.588385 10129 raft_consensus.cc:2932] T e7a8fc5afcab4b1db4ebf245b66df4be P 4b7f7ba2ac084e829c4304c16a94894f [term 3 FOLLOWER]: Committing config change with OpId 3.29: config changed from index 26 to 29, ae04d48c97204162b0f8a27d81fcec1a (127.6.117.198) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } attrs { replace: true } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "ae04d48c97204162b0f8a27d81fcec1a" member_type: VOTER last_known_addr { host: "127.6.117.198" port: 33655 } attrs { promote: false } } }
I20240502 01:59:35.588449 9947 catalog_manager.cc:5484] T e7a8fc5afcab4b1db4ebf245b66df4be P ae04d48c97204162b0f8a27d81fcec1a reported cstate change: config changed from index 26 to 29, ae04d48c97204162b0f8a27d81fcec1a (127.6.117.198) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } attrs { replace: true } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "ae04d48c97204162b0f8a27d81fcec1a" member_type: VOTER last_known_addr { host: "127.6.117.198" port: 33655 } attrs { promote: false } } }
I20240502 01:59:35.588567 10718 raft_consensus.cc:2932] T b4409766ce994f698c31aede604a6927 P 4b7f7ba2ac084e829c4304c16a94894f [term 3 LEADER]: Committing config change with OpId 3.29: config changed from index 28 to 29, VOTER bdbecaeb8f214529b2012a9c003de2ff (127.6.117.193) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "ae04d48c97204162b0f8a27d81fcec1a" member_type: VOTER last_known_addr { host: "127.6.117.198" port: 33655 } attrs { promote: false } } }
I20240502 01:59:35.589023 10054 raft_consensus.cc:2932] T e7a8fc5afcab4b1db4ebf245b66df4be P bdbecaeb8f214529b2012a9c003de2ff [term 3 FOLLOWER]: Committing config change with OpId 3.29: config changed from index 26 to 29, ae04d48c97204162b0f8a27d81fcec1a (127.6.117.198) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } attrs { replace: true } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "ae04d48c97204162b0f8a27d81fcec1a" member_type: VOTER last_known_addr { host: "127.6.117.198" port: 33655 } attrs { promote: false } } }
I20240502 01:59:35.590651 9940 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b4409766ce994f698c31aede604a6927 with cas_config_opid_index 28: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:59:35.591059 10200 raft_consensus.cc:2932] T b4409766ce994f698c31aede604a6927 P e77d1c0d41034d1eb5a036dc17c071f5 [term 3 FOLLOWER]: Committing config change with OpId 3.29: config changed from index 28 to 29, VOTER bdbecaeb8f214529b2012a9c003de2ff (127.6.117.193) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "ae04d48c97204162b0f8a27d81fcec1a" member_type: VOTER last_known_addr { host: "127.6.117.198" port: 33655 } attrs { promote: false } } }
W20240502 01:59:35.592581 9940 catalog_manager.cc:5007] ChangeConfig:REMOVE_PEER RPC for tablet b4409766ce994f698c31aede604a6927 with cas_config_opid_index 28: ChangeConfig:REMOVE_PEER failed with leader 4b7f7ba2ac084e829c4304c16a94894f (127.6.117.194:46805) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 28 but the committed config has opid_index of 29
I20240502 01:59:35.593566 10590 raft_consensus.cc:2932] T b4409766ce994f698c31aede604a6927 P ae04d48c97204162b0f8a27d81fcec1a [term 3 FOLLOWER]: Committing config change with OpId 3.29: config changed from index 28 to 29, VOTER bdbecaeb8f214529b2012a9c003de2ff (127.6.117.193) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "ae04d48c97204162b0f8a27d81fcec1a" member_type: VOTER last_known_addr { host: "127.6.117.198" port: 33655 } attrs { promote: false } } }
I20240502 01:59:35.595022 9948 catalog_manager.cc:5484] T b4409766ce994f698c31aede604a6927 P 4b7f7ba2ac084e829c4304c16a94894f reported cstate change: config changed from index 28 to 29, VOTER bdbecaeb8f214529b2012a9c003de2ff (127.6.117.193) evicted. New cstate: current_term: 3 leader_uuid: "4b7f7ba2ac084e829c4304c16a94894f" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae04d48c97204162b0f8a27d81fcec1a" member_type: VOTER last_known_addr { host: "127.6.117.198" port: 33655 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:59:35.595952 10200 consensus_queue.cc:237] T e7a8fc5afcab4b1db4ebf245b66df4be P e77d1c0d41034d1eb5a036dc17c071f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 3.29, Last appended by leader: 27, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "ae04d48c97204162b0f8a27d81fcec1a" member_type: VOTER last_known_addr { host: "127.6.117.198" port: 33655 } attrs { promote: false } }
I20240502 01:59:35.597453 10044 tablet_service.cc:1514] Processing DeleteTablet for tablet b4409766ce994f698c31aede604a6927 with delete_type TABLET_DATA_TOMBSTONED (TS bdbecaeb8f214529b2012a9c003de2ff not found in new config with opid_index 29) from {username='slave'} at 127.0.0.1:44390
I20240502 01:59:35.598129 10726 tablet_replica.cc:330] T b4409766ce994f698c31aede604a6927 P bdbecaeb8f214529b2012a9c003de2ff: stopping tablet replica
I20240502 01:59:35.598366 10726 raft_consensus.cc:2229] T b4409766ce994f698c31aede604a6927 P bdbecaeb8f214529b2012a9c003de2ff [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:35.598651 10054 raft_consensus.cc:1261] T e7a8fc5afcab4b1db4ebf245b66df4be P bdbecaeb8f214529b2012a9c003de2ff [term 3 FOLLOWER]: Refusing update from remote peer e77d1c0d41034d1eb5a036dc17c071f5: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240502 01:59:35.598659 10590 raft_consensus.cc:1261] T e7a8fc5afcab4b1db4ebf245b66df4be P ae04d48c97204162b0f8a27d81fcec1a [term 3 FOLLOWER]: Refusing update from remote peer e77d1c0d41034d1eb5a036dc17c071f5: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240502 01:59:35.599370 10726 raft_consensus.cc:2258] T b4409766ce994f698c31aede604a6927 P bdbecaeb8f214529b2012a9c003de2ff [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:35.599447 10722 consensus_queue.cc:1035] T e7a8fc5afcab4b1db4ebf245b66df4be P e77d1c0d41034d1eb5a036dc17c071f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 01:59:35.599763 10722 consensus_queue.cc:1035] T e7a8fc5afcab4b1db4ebf245b66df4be P e77d1c0d41034d1eb5a036dc17c071f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae04d48c97204162b0f8a27d81fcec1a" member_type: VOTER last_known_addr { host: "127.6.117.198" port: 33655 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 01:59:35.600832 10726 ts_tablet_manager.cc:1875] T b4409766ce994f698c31aede604a6927 P bdbecaeb8f214529b2012a9c003de2ff: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:59:35.601439 10702 raft_consensus.cc:2932] T e7a8fc5afcab4b1db4ebf245b66df4be P e77d1c0d41034d1eb5a036dc17c071f5 [term 3 LEADER]: Committing config change with OpId 3.30: config changed from index 29 to 30, VOTER 4b7f7ba2ac084e829c4304c16a94894f (127.6.117.194) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "ae04d48c97204162b0f8a27d81fcec1a" member_type: VOTER last_known_addr { host: "127.6.117.198" port: 33655 } attrs { promote: false } } }
I20240502 01:59:35.601790 10054 raft_consensus.cc:2932] T e7a8fc5afcab4b1db4ebf245b66df4be P bdbecaeb8f214529b2012a9c003de2ff [term 3 FOLLOWER]: Committing config change with OpId 3.30: config changed from index 29 to 30, VOTER 4b7f7ba2ac084e829c4304c16a94894f (127.6.117.194) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "ae04d48c97204162b0f8a27d81fcec1a" member_type: VOTER last_known_addr { host: "127.6.117.198" port: 33655 } attrs { promote: false } } }
I20240502 01:59:35.602917 9939 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e7a8fc5afcab4b1db4ebf245b66df4be with cas_config_opid_index 29: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:59:35.603701 9953 catalog_manager.cc:5484] T e7a8fc5afcab4b1db4ebf245b66df4be P e77d1c0d41034d1eb5a036dc17c071f5 reported cstate change: config changed from index 29 to 30, VOTER 4b7f7ba2ac084e829c4304c16a94894f (127.6.117.194) evicted. New cstate: current_term: 3 leader_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae04d48c97204162b0f8a27d81fcec1a" member_type: VOTER last_known_addr { host: "127.6.117.198" port: 33655 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:59:35.603770 10590 raft_consensus.cc:2932] T e7a8fc5afcab4b1db4ebf245b66df4be P ae04d48c97204162b0f8a27d81fcec1a [term 3 FOLLOWER]: Committing config change with OpId 3.30: config changed from index 29 to 30, VOTER 4b7f7ba2ac084e829c4304c16a94894f (127.6.117.194) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "ae04d48c97204162b0f8a27d81fcec1a" member_type: VOTER last_known_addr { host: "127.6.117.198" port: 33655 } attrs { promote: false } } }
I20240502 01:59:35.604485 10726 ts_tablet_manager.cc:1888] T b4409766ce994f698c31aede604a6927 P bdbecaeb8f214529b2012a9c003de2ff: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.28
I20240502 01:59:35.604552 10726 log.cc:1196] T b4409766ce994f698c31aede604a6927 P bdbecaeb8f214529b2012a9c003de2ff: Deleting WAL directory at /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714615076189642-6615-0/minicluster-data/ts-0-root/wals/b4409766ce994f698c31aede604a6927
I20240502 01:59:35.605214 9941 catalog_manager.cc:4830] TS bdbecaeb8f214529b2012a9c003de2ff (127.6.117.193:34597): tablet b4409766ce994f698c31aede604a6927 (table test-workload [id=10a74377088c4a9890835737184010db]) successfully deleted
I20240502 01:59:35.607785 10116 tablet_service.cc:1514] Processing DeleteTablet for tablet e7a8fc5afcab4b1db4ebf245b66df4be with delete_type TABLET_DATA_TOMBSTONED (TS 4b7f7ba2ac084e829c4304c16a94894f not found in new config with opid_index 30) from {username='slave'} at 127.0.0.1:52598
I20240502 01:59:35.612741 10727 tablet_replica.cc:330] T e7a8fc5afcab4b1db4ebf245b66df4be P 4b7f7ba2ac084e829c4304c16a94894f: stopping tablet replica
I20240502 01:59:35.613039 10727 raft_consensus.cc:2229] T e7a8fc5afcab4b1db4ebf245b66df4be P 4b7f7ba2ac084e829c4304c16a94894f [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:35.613548 10727 raft_consensus.cc:2258] T e7a8fc5afcab4b1db4ebf245b66df4be P 4b7f7ba2ac084e829c4304c16a94894f [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:35.614180 10666 tablet_bootstrap.cc:492] T 1851ea6467584b0ea32a1f68696803d0 P d6ff17402670432da6eeb5da2af7b4b6: Bootstrap replayed 1/1 log segments. Stats: ops{read=26 overwritten=0 applied=26 ignored=0} inserts{seen=80 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:59:35.614531 10727 ts_tablet_manager.cc:1875] T e7a8fc5afcab4b1db4ebf245b66df4be P 4b7f7ba2ac084e829c4304c16a94894f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:59:35.614550 10666 tablet_bootstrap.cc:492] T 1851ea6467584b0ea32a1f68696803d0 P d6ff17402670432da6eeb5da2af7b4b6: Bootstrap complete.
I20240502 01:59:35.614943 10666 ts_tablet_manager.cc:1367] T 1851ea6467584b0ea32a1f68696803d0 P d6ff17402670432da6eeb5da2af7b4b6: Time spent bootstrapping tablet: real 0.058s user 0.018s sys 0.006s
I20240502 01:59:35.615525 10666 raft_consensus.cc:348] T 1851ea6467584b0ea32a1f68696803d0 P d6ff17402670432da6eeb5da2af7b4b6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "d6ff17402670432da6eeb5da2af7b4b6" member_type: NON_VOTER last_known_addr { host: "127.6.117.196" port: 34337 } attrs { promote: true } }
I20240502 01:59:35.615644 10666 raft_consensus.cc:729] T 1851ea6467584b0ea32a1f68696803d0 P d6ff17402670432da6eeb5da2af7b4b6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d6ff17402670432da6eeb5da2af7b4b6, State: Initialized, Role: LEARNER
I20240502 01:59:35.615760 10666 consensus_queue.cc:260] T 1851ea6467584b0ea32a1f68696803d0 P d6ff17402670432da6eeb5da2af7b4b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "d6ff17402670432da6eeb5da2af7b4b6" member_type: NON_VOTER last_known_addr { host: "127.6.117.196" port: 34337 } attrs { promote: true } }
I20240502 01:59:35.616658 10666 ts_tablet_manager.cc:1398] T 1851ea6467584b0ea32a1f68696803d0 P d6ff17402670432da6eeb5da2af7b4b6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:59:35.617097 10143 tablet_copy_service.cc:342] P 4b7f7ba2ac084e829c4304c16a94894f: Request end of tablet copy session d6ff17402670432da6eeb5da2af7b4b6-1851ea6467584b0ea32a1f68696803d0 received from {username='slave'} at 127.0.0.1:52658
I20240502 01:59:35.617218 10143 tablet_copy_service.cc:433] P 4b7f7ba2ac084e829c4304c16a94894f: ending tablet copy session d6ff17402670432da6eeb5da2af7b4b6-1851ea6467584b0ea32a1f68696803d0 on tablet 1851ea6467584b0ea32a1f68696803d0 with peer d6ff17402670432da6eeb5da2af7b4b6
I20240502 01:59:35.617777 10727 ts_tablet_manager.cc:1888] T e7a8fc5afcab4b1db4ebf245b66df4be P 4b7f7ba2ac084e829c4304c16a94894f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.29
I20240502 01:59:35.618098 10727 log.cc:1196] T e7a8fc5afcab4b1db4ebf245b66df4be P 4b7f7ba2ac084e829c4304c16a94894f: Deleting WAL directory at /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714615076189642-6615-0/minicluster-data/ts-1-root/wals/e7a8fc5afcab4b1db4ebf245b66df4be
I20240502 01:59:35.618964 9940 catalog_manager.cc:4830] TS 4b7f7ba2ac084e829c4304c16a94894f (127.6.117.194:46805): tablet e7a8fc5afcab4b1db4ebf245b66df4be (table test-workload [id=10a74377088c4a9890835737184010db]) successfully deleted
I20240502 01:59:35.619599 10671 tablet_bootstrap.cc:492] T 19ff9c216a094fde86ea9f9648bf08cb P d52d70d7578f4995b03ac6faba3ec739: Bootstrap replayed 1/1 log segments. Stats: ops{read=25 overwritten=0 applied=25 ignored=0} inserts{seen=102 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:59:35.619843 10671 tablet_bootstrap.cc:492] T 19ff9c216a094fde86ea9f9648bf08cb P d52d70d7578f4995b03ac6faba3ec739: Bootstrap complete.
I20240502 01:59:35.620010 10671 ts_tablet_manager.cc:1367] T 19ff9c216a094fde86ea9f9648bf08cb P d52d70d7578f4995b03ac6faba3ec739: Time spent bootstrapping tablet: real 0.045s user 0.016s sys 0.005s
I20240502 01:59:35.620494 10671 raft_consensus.cc:348] T 19ff9c216a094fde86ea9f9648bf08cb P d52d70d7578f4995b03ac6faba3ec739 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } attrs { replace: true } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "d52d70d7578f4995b03ac6faba3ec739" member_type: NON_VOTER last_known_addr { host: "127.6.117.197" port: 44467 } attrs { promote: true } }
I20240502 01:59:35.620626 10671 raft_consensus.cc:729] T 19ff9c216a094fde86ea9f9648bf08cb P d52d70d7578f4995b03ac6faba3ec739 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d52d70d7578f4995b03ac6faba3ec739, State: Initialized, Role: LEARNER
I20240502 01:59:35.620730 10671 consensus_queue.cc:260] T 19ff9c216a094fde86ea9f9648bf08cb P d52d70d7578f4995b03ac6faba3ec739 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } attrs { replace: true } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "d52d70d7578f4995b03ac6faba3ec739" member_type: NON_VOTER last_known_addr { host: "127.6.117.197" port: 44467 } attrs { promote: true } }
I20240502 01:59:35.620993 10671 ts_tablet_manager.cc:1398] T 19ff9c216a094fde86ea9f9648bf08cb P d52d70d7578f4995b03ac6faba3ec739: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240502 01:59:35.621435 10143 tablet_copy_service.cc:342] P 4b7f7ba2ac084e829c4304c16a94894f: Request end of tablet copy session d52d70d7578f4995b03ac6faba3ec739-19ff9c216a094fde86ea9f9648bf08cb received from {username='slave'} at 127.0.0.1:52672
I20240502 01:59:35.621507 10143 tablet_copy_service.cc:433] P 4b7f7ba2ac084e829c4304c16a94894f: ending tablet copy session d52d70d7578f4995b03ac6faba3ec739-19ff9c216a094fde86ea9f9648bf08cb on tablet 19ff9c216a094fde86ea9f9648bf08cb with peer d52d70d7578f4995b03ac6faba3ec739
I20240502 01:59:35.720948 10441 raft_consensus.cc:1203] T 1851ea6467584b0ea32a1f68696803d0 P d6ff17402670432da6eeb5da2af7b4b6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.25->[1.26-1.26] Dedup: 1.26->[]
I20240502 01:59:35.763118 10441 raft_consensus.cc:1203] T 73ea0bfb40ad43a68a6289ff1b35d005 P d6ff17402670432da6eeb5da2af7b4b6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.24->[1.25-1.25] Dedup: 1.25->[]
I20240502 01:59:35.898983 10517 raft_consensus.cc:1203] T 580db11a61cb46f48d5f7507cfbdbb52 P d52d70d7578f4995b03ac6faba3ec739 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.25->[1.26-1.26] Dedup: 1.26->[]
I20240502 01:59:35.960880 10008 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:59:35.962502 10129 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1851ea6467584b0ea32a1f68696803d0"
dest_uuid: "4b7f7ba2ac084e829c4304c16a94894f"
mode: GRACEFUL
new_leader_uuid: "bdbecaeb8f214529b2012a9c003de2ff"
from {username='slave'} at 127.0.0.1:52630
I20240502 01:59:35.962692 10129 raft_consensus.cc:595] T 1851ea6467584b0ea32a1f68696803d0 P 4b7f7ba2ac084e829c4304c16a94894f [term 1 LEADER]: Received request to transfer leadership to TS bdbecaeb8f214529b2012a9c003de2ff
I20240502 01:59:35.963433 10200 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e7a8fc5afcab4b1db4ebf245b66df4be"
dest_uuid: "e77d1c0d41034d1eb5a036dc17c071f5"
mode: GRACEFUL
new_leader_uuid: "ae04d48c97204162b0f8a27d81fcec1a"
from {username='slave'} at 127.0.0.1:48104
I20240502 01:59:35.963570 10200 raft_consensus.cc:595] T e7a8fc5afcab4b1db4ebf245b66df4be P e77d1c0d41034d1eb5a036dc17c071f5 [term 3 LEADER]: Received request to transfer leadership to TS ae04d48c97204162b0f8a27d81fcec1a
I20240502 01:59:35.963853 10008 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 2
I20240502 01:59:35.964000 10008 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:59:35.992537 10722 raft_consensus.cc:979] T e7a8fc5afcab4b1db4ebf245b66df4be P e77d1c0d41034d1eb5a036dc17c071f5: : Instructing follower ae04d48c97204162b0f8a27d81fcec1a to start an election
I20240502 01:59:35.992686 10725 raft_consensus.cc:1067] T e7a8fc5afcab4b1db4ebf245b66df4be P e77d1c0d41034d1eb5a036dc17c071f5 [term 3 LEADER]: Signalling peer ae04d48c97204162b0f8a27d81fcec1a to start an election
I20240502 01:59:35.993153 10590 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e7a8fc5afcab4b1db4ebf245b66df4be"
dest_uuid: "ae04d48c97204162b0f8a27d81fcec1a"
from {username='slave'} at 127.0.0.1:36526
I20240502 01:59:35.993307 10590 raft_consensus.cc:482] T e7a8fc5afcab4b1db4ebf245b66df4be P ae04d48c97204162b0f8a27d81fcec1a [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:59:35.993361 10590 raft_consensus.cc:3037] T e7a8fc5afcab4b1db4ebf245b66df4be P ae04d48c97204162b0f8a27d81fcec1a [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:59:35.994269 10590 raft_consensus.cc:504] T e7a8fc5afcab4b1db4ebf245b66df4be P ae04d48c97204162b0f8a27d81fcec1a [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "ae04d48c97204162b0f8a27d81fcec1a" member_type: VOTER last_known_addr { host: "127.6.117.198" port: 33655 } attrs { promote: false } }
I20240502 01:59:35.994731 10590 leader_election.cc:288] T e7a8fc5afcab4b1db4ebf245b66df4be P ae04d48c97204162b0f8a27d81fcec1a [CANDIDATE]: Term 4 election: Requested vote from peers bdbecaeb8f214529b2012a9c003de2ff (127.6.117.193:34597), e77d1c0d41034d1eb5a036dc17c071f5 (127.6.117.195:43977)
I20240502 01:59:35.995129 10054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7a8fc5afcab4b1db4ebf245b66df4be" candidate_uuid: "ae04d48c97204162b0f8a27d81fcec1a" candidate_term: 4 candidate_status { last_received { term: 3 index: 30 } } ignore_live_leader: true dest_uuid: "bdbecaeb8f214529b2012a9c003de2ff"
I20240502 01:59:35.995282 10054 raft_consensus.cc:3037] T e7a8fc5afcab4b1db4ebf245b66df4be P bdbecaeb8f214529b2012a9c003de2ff [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:59:35.996053 10054 raft_consensus.cc:2454] T e7a8fc5afcab4b1db4ebf245b66df4be P bdbecaeb8f214529b2012a9c003de2ff [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae04d48c97204162b0f8a27d81fcec1a in term 4.
I20240502 01:59:35.995131 10200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7a8fc5afcab4b1db4ebf245b66df4be" candidate_uuid: "ae04d48c97204162b0f8a27d81fcec1a" candidate_term: 4 candidate_status { last_received { term: 3 index: 30 } } ignore_live_leader: true dest_uuid: "e77d1c0d41034d1eb5a036dc17c071f5"
I20240502 01:59:35.996344 10200 raft_consensus.cc:3032] T e7a8fc5afcab4b1db4ebf245b66df4be P e77d1c0d41034d1eb5a036dc17c071f5 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 01:59:35.996384 10200 raft_consensus.cc:729] T e7a8fc5afcab4b1db4ebf245b66df4be P e77d1c0d41034d1eb5a036dc17c071f5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: e77d1c0d41034d1eb5a036dc17c071f5, State: Running, Role: LEADER
I20240502 01:59:35.996399 10558 leader_election.cc:302] T e7a8fc5afcab4b1db4ebf245b66df4be P ae04d48c97204162b0f8a27d81fcec1a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae04d48c97204162b0f8a27d81fcec1a, bdbecaeb8f214529b2012a9c003de2ff; no voters:
I20240502 01:59:35.996546 10200 consensus_queue.cc:260] T e7a8fc5afcab4b1db4ebf245b66df4be P e77d1c0d41034d1eb5a036dc17c071f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "ae04d48c97204162b0f8a27d81fcec1a" member_type: VOTER last_known_addr { host: "127.6.117.198" port: 33655 } attrs { promote: false } }
I20240502 01:59:35.996613 10717 raft_consensus.cc:2781] T e7a8fc5afcab4b1db4ebf245b66df4be P ae04d48c97204162b0f8a27d81fcec1a [term 4 FOLLOWER]: Leader election won for term 4
I20240502 01:59:35.996708 10717 raft_consensus.cc:686] T e7a8fc5afcab4b1db4ebf245b66df4be P ae04d48c97204162b0f8a27d81fcec1a [term 4 LEADER]: Becoming Leader. State: Replica: ae04d48c97204162b0f8a27d81fcec1a, State: Running, Role: LEADER
I20240502 01:59:35.996729 10200 raft_consensus.cc:3037] T e7a8fc5afcab4b1db4ebf245b66df4be P e77d1c0d41034d1eb5a036dc17c071f5 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:59:35.996848 10717 consensus_queue.cc:237] T e7a8fc5afcab4b1db4ebf245b66df4be P ae04d48c97204162b0f8a27d81fcec1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "ae04d48c97204162b0f8a27d81fcec1a" member_type: VOTER last_known_addr { host: "127.6.117.198" port: 33655 } attrs { promote: false } }
I20240502 01:59:35.997498 10200 raft_consensus.cc:2454] T e7a8fc5afcab4b1db4ebf245b66df4be P e77d1c0d41034d1eb5a036dc17c071f5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae04d48c97204162b0f8a27d81fcec1a in term 4.
I20240502 01:59:35.998407 9948 catalog_manager.cc:5484] T e7a8fc5afcab4b1db4ebf245b66df4be P ae04d48c97204162b0f8a27d81fcec1a reported cstate change: term changed from 3 to 4, leader changed from e77d1c0d41034d1eb5a036dc17c071f5 (127.6.117.195) to ae04d48c97204162b0f8a27d81fcec1a (127.6.117.198). New cstate: current_term: 4 leader_uuid: "ae04d48c97204162b0f8a27d81fcec1a" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae04d48c97204162b0f8a27d81fcec1a" member_type: VOTER last_known_addr { host: "127.6.117.198" port: 33655 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:59:36.047106 10517 raft_consensus.cc:1203] T 19ff9c216a094fde86ea9f9648bf08cb P d52d70d7578f4995b03ac6faba3ec739 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.24->[1.25-1.25] Dedup: 1.25->[]
I20240502 01:59:36.129665 10728 raft_consensus.cc:1050] T 1851ea6467584b0ea32a1f68696803d0 P 4b7f7ba2ac084e829c4304c16a94894f: attempting to promote NON_VOTER d6ff17402670432da6eeb5da2af7b4b6 to VOTER
W20240502 01:59:36.129791 10728 raft_consensus.cc:1053] T 1851ea6467584b0ea32a1f68696803d0 P 4b7f7ba2ac084e829c4304c16a94894f: Unable to promote non-voter d6ff17402670432da6eeb5da2af7b4b6: Service unavailable: leader transfer in progress
I20240502 01:59:36.231420 10728 raft_consensus.cc:1050] T 73ea0bfb40ad43a68a6289ff1b35d005 P 4b7f7ba2ac084e829c4304c16a94894f: attempting to promote NON_VOTER d6ff17402670432da6eeb5da2af7b4b6 to VOTER
I20240502 01:59:36.232064 10728 consensus_queue.cc:237] T 73ea0bfb40ad43a68a6289ff1b35d005 P 4b7f7ba2ac084e829c4304c16a94894f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "d6ff17402670432da6eeb5da2af7b4b6" member_type: VOTER last_known_addr { host: "127.6.117.196" port: 34337 } attrs { promote: false } }
I20240502 01:59:36.233883 10200 raft_consensus.cc:1261] T 73ea0bfb40ad43a68a6289ff1b35d005 P e77d1c0d41034d1eb5a036dc17c071f5 [term 1 FOLLOWER]: Refusing update from remote peer 4b7f7ba2ac084e829c4304c16a94894f: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240502 01:59:36.234314 10441 raft_consensus.cc:1261] T 73ea0bfb40ad43a68a6289ff1b35d005 P d6ff17402670432da6eeb5da2af7b4b6 [term 1 LEARNER]: Refusing update from remote peer 4b7f7ba2ac084e829c4304c16a94894f: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240502 01:59:36.234283 10731 consensus_queue.cc:1035] T 73ea0bfb40ad43a68a6289ff1b35d005 P 4b7f7ba2ac084e829c4304c16a94894f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 01:59:36.234740 10053 raft_consensus.cc:1261] T 73ea0bfb40ad43a68a6289ff1b35d005 P bdbecaeb8f214529b2012a9c003de2ff [term 1 FOLLOWER]: Refusing update from remote peer 4b7f7ba2ac084e829c4304c16a94894f: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240502 01:59:36.234688 10677 consensus_queue.cc:1035] T 73ea0bfb40ad43a68a6289ff1b35d005 P 4b7f7ba2ac084e829c4304c16a94894f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6ff17402670432da6eeb5da2af7b4b6" member_type: VOTER last_known_addr { host: "127.6.117.196" port: 34337 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 01:59:36.235090 10677 consensus_queue.cc:1035] T 73ea0bfb40ad43a68a6289ff1b35d005 P 4b7f7ba2ac084e829c4304c16a94894f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 01:59:36.236910 10729 raft_consensus.cc:2932] T 73ea0bfb40ad43a68a6289ff1b35d005 P 4b7f7ba2ac084e829c4304c16a94894f [term 1 LEADER]: Committing config change with OpId 1.26: config changed from index 25 to 26, d6ff17402670432da6eeb5da2af7b4b6 (127.6.117.196) changed from NON_VOTER to VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "d6ff17402670432da6eeb5da2af7b4b6" member_type: VOTER last_known_addr { host: "127.6.117.196" port: 34337 } attrs { promote: false } } }
I20240502 01:59:36.237581 10441 raft_consensus.cc:2932] T 73ea0bfb40ad43a68a6289ff1b35d005 P d6ff17402670432da6eeb5da2af7b4b6 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index 25 to 26, d6ff17402670432da6eeb5da2af7b4b6 (127.6.117.196) changed from NON_VOTER to VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "d6ff17402670432da6eeb5da2af7b4b6" member_type: VOTER last_known_addr { host: "127.6.117.196" port: 34337 } attrs { promote: false } } }
I20240502 01:59:36.239290 10053 raft_consensus.cc:2932] T 73ea0bfb40ad43a68a6289ff1b35d005 P bdbecaeb8f214529b2012a9c003de2ff [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index 25 to 26, d6ff17402670432da6eeb5da2af7b4b6 (127.6.117.196) changed from NON_VOTER to VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "d6ff17402670432da6eeb5da2af7b4b6" member_type: VOTER last_known_addr { host: "127.6.117.196" port: 34337 } attrs { promote: false } } }
I20240502 01:59:36.239827 10200 raft_consensus.cc:2932] T 73ea0bfb40ad43a68a6289ff1b35d005 P e77d1c0d41034d1eb5a036dc17c071f5 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index 25 to 26, d6ff17402670432da6eeb5da2af7b4b6 (127.6.117.196) changed from NON_VOTER to VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "d6ff17402670432da6eeb5da2af7b4b6" member_type: VOTER last_known_addr { host: "127.6.117.196" port: 34337 } attrs { promote: false } } }
I20240502 01:59:36.241139 9948 catalog_manager.cc:5484] T 73ea0bfb40ad43a68a6289ff1b35d005 P d6ff17402670432da6eeb5da2af7b4b6 reported cstate change: config changed from index 25 to 26, d6ff17402670432da6eeb5da2af7b4b6 (127.6.117.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4b7f7ba2ac084e829c4304c16a94894f" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "d6ff17402670432da6eeb5da2af7b4b6" member_type: VOTER last_known_addr { host: "127.6.117.196" port: 34337 } attrs { promote: false } } }
I20240502 01:59:36.244518 10131 consensus_queue.cc:237] T 73ea0bfb40ad43a68a6289ff1b35d005 P 4b7f7ba2ac084e829c4304c16a94894f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "d6ff17402670432da6eeb5da2af7b4b6" member_type: VOTER last_known_addr { host: "127.6.117.196" port: 34337 } attrs { promote: false } }
I20240502 01:59:36.245545 10053 raft_consensus.cc:1261] T 73ea0bfb40ad43a68a6289ff1b35d005 P bdbecaeb8f214529b2012a9c003de2ff [term 1 FOLLOWER]: Refusing update from remote peer 4b7f7ba2ac084e829c4304c16a94894f: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240502 01:59:36.245978 10441 raft_consensus.cc:1261] T 73ea0bfb40ad43a68a6289ff1b35d005 P d6ff17402670432da6eeb5da2af7b4b6 [term 1 FOLLOWER]: Refusing update from remote peer 4b7f7ba2ac084e829c4304c16a94894f: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240502 01:59:36.246050 10728 consensus_queue.cc:1035] T 73ea0bfb40ad43a68a6289ff1b35d005 P 4b7f7ba2ac084e829c4304c16a94894f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 01:59:36.246433 10728 consensus_queue.cc:1035] T 73ea0bfb40ad43a68a6289ff1b35d005 P 4b7f7ba2ac084e829c4304c16a94894f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6ff17402670432da6eeb5da2af7b4b6" member_type: VOTER last_known_addr { host: "127.6.117.196" port: 34337 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 01:59:36.247592 10731 raft_consensus.cc:2932] T 73ea0bfb40ad43a68a6289ff1b35d005 P 4b7f7ba2ac084e829c4304c16a94894f [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index 26 to 27, VOTER e77d1c0d41034d1eb5a036dc17c071f5 (127.6.117.195) evicted. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "d6ff17402670432da6eeb5da2af7b4b6" member_type: VOTER last_known_addr { host: "127.6.117.196" port: 34337 } attrs { promote: false } } }
I20240502 01:59:36.248332 10053 raft_consensus.cc:2932] T 73ea0bfb40ad43a68a6289ff1b35d005 P bdbecaeb8f214529b2012a9c003de2ff [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, VOTER e77d1c0d41034d1eb5a036dc17c071f5 (127.6.117.195) evicted. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "d6ff17402670432da6eeb5da2af7b4b6" member_type: VOTER last_known_addr { host: "127.6.117.196" port: 34337 } attrs { promote: false } } }
I20240502 01:59:36.249167 9940 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 73ea0bfb40ad43a68a6289ff1b35d005 with cas_config_opid_index 26: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:59:36.249934 10441 raft_consensus.cc:2932] T 73ea0bfb40ad43a68a6289ff1b35d005 P d6ff17402670432da6eeb5da2af7b4b6 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, VOTER e77d1c0d41034d1eb5a036dc17c071f5 (127.6.117.195) evicted. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "d6ff17402670432da6eeb5da2af7b4b6" member_type: VOTER last_known_addr { host: "127.6.117.196" port: 34337 } attrs { promote: false } } }
I20240502 01:59:36.250154 9953 catalog_manager.cc:5484] T 73ea0bfb40ad43a68a6289ff1b35d005 P 4b7f7ba2ac084e829c4304c16a94894f reported cstate change: config changed from index 26 to 27, VOTER e77d1c0d41034d1eb5a036dc17c071f5 (127.6.117.195) evicted. New cstate: current_term: 1 leader_uuid: "4b7f7ba2ac084e829c4304c16a94894f" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6ff17402670432da6eeb5da2af7b4b6" member_type: VOTER last_known_addr { host: "127.6.117.196" port: 34337 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:59:36.252391 10192 tablet_service.cc:1514] Processing DeleteTablet for tablet 73ea0bfb40ad43a68a6289ff1b35d005 with delete_type TABLET_DATA_TOMBSTONED (TS e77d1c0d41034d1eb5a036dc17c071f5 not found in new config with opid_index 27) from {username='slave'} at 127.0.0.1:48060
I20240502 01:59:36.252841 10735 tablet_replica.cc:330] T 73ea0bfb40ad43a68a6289ff1b35d005 P e77d1c0d41034d1eb5a036dc17c071f5: stopping tablet replica
I20240502 01:59:36.253016 10735 raft_consensus.cc:2229] T 73ea0bfb40ad43a68a6289ff1b35d005 P e77d1c0d41034d1eb5a036dc17c071f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:36.253240 10735 raft_consensus.cc:2258] T 73ea0bfb40ad43a68a6289ff1b35d005 P e77d1c0d41034d1eb5a036dc17c071f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:36.253844 10735 ts_tablet_manager.cc:1875] T 73ea0bfb40ad43a68a6289ff1b35d005 P e77d1c0d41034d1eb5a036dc17c071f5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:59:36.255976 10735 ts_tablet_manager.cc:1888] T 73ea0bfb40ad43a68a6289ff1b35d005 P e77d1c0d41034d1eb5a036dc17c071f5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.26
I20240502 01:59:36.256071 10735 log.cc:1196] T 73ea0bfb40ad43a68a6289ff1b35d005 P e77d1c0d41034d1eb5a036dc17c071f5: Deleting WAL directory at /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714615076189642-6615-0/minicluster-data/ts-2-root/wals/73ea0bfb40ad43a68a6289ff1b35d005
I20240502 01:59:36.256641 9939 catalog_manager.cc:4830] TS e77d1c0d41034d1eb5a036dc17c071f5 (127.6.117.195:43977): tablet 73ea0bfb40ad43a68a6289ff1b35d005 (table test-workload [id=10a74377088c4a9890835737184010db]) successfully deleted
I20240502 01:59:36.409848 10728 raft_consensus.cc:979] T 1851ea6467584b0ea32a1f68696803d0 P 4b7f7ba2ac084e829c4304c16a94894f: : Instructing follower bdbecaeb8f214529b2012a9c003de2ff to start an election
I20240502 01:59:36.409984 10728 raft_consensus.cc:1067] T 1851ea6467584b0ea32a1f68696803d0 P 4b7f7ba2ac084e829c4304c16a94894f [term 1 LEADER]: Signalling peer bdbecaeb8f214529b2012a9c003de2ff to start an election
I20240502 01:59:36.410377 10053 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1851ea6467584b0ea32a1f68696803d0"
dest_uuid: "bdbecaeb8f214529b2012a9c003de2ff"
from {username='slave'} at 127.0.0.1:44396
I20240502 01:59:36.410523 10053 raft_consensus.cc:482] T 1851ea6467584b0ea32a1f68696803d0 P bdbecaeb8f214529b2012a9c003de2ff [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:59:36.410595 10053 raft_consensus.cc:3037] T 1851ea6467584b0ea32a1f68696803d0 P bdbecaeb8f214529b2012a9c003de2ff [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:59:36.411465 10053 raft_consensus.cc:504] T 1851ea6467584b0ea32a1f68696803d0 P bdbecaeb8f214529b2012a9c003de2ff [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "d6ff17402670432da6eeb5da2af7b4b6" member_type: NON_VOTER last_known_addr { host: "127.6.117.196" port: 34337 } attrs { promote: true } }
I20240502 01:59:36.411798 10053 leader_election.cc:288] T 1851ea6467584b0ea32a1f68696803d0 P bdbecaeb8f214529b2012a9c003de2ff [CANDIDATE]: Term 2 election: Requested vote from peers 4b7f7ba2ac084e829c4304c16a94894f (127.6.117.194:46805), e77d1c0d41034d1eb5a036dc17c071f5 (127.6.117.195:43977)
I20240502 01:59:36.412096 10200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1851ea6467584b0ea32a1f68696803d0" candidate_uuid: "bdbecaeb8f214529b2012a9c003de2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: true dest_uuid: "e77d1c0d41034d1eb5a036dc17c071f5"
I20240502 01:59:36.412214 10200 raft_consensus.cc:3037] T 1851ea6467584b0ea32a1f68696803d0 P e77d1c0d41034d1eb5a036dc17c071f5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:59:36.413126 10131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1851ea6467584b0ea32a1f68696803d0" candidate_uuid: "bdbecaeb8f214529b2012a9c003de2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: true dest_uuid: "4b7f7ba2ac084e829c4304c16a94894f"
I20240502 01:59:36.413309 10131 raft_consensus.cc:3032] T 1851ea6467584b0ea32a1f68696803d0 P 4b7f7ba2ac084e829c4304c16a94894f [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:59:36.413339 10706 raft_consensus.cc:1050] T 580db11a61cb46f48d5f7507cfbdbb52 P bdbecaeb8f214529b2012a9c003de2ff: attempting to promote NON_VOTER d52d70d7578f4995b03ac6faba3ec739 to VOTER
I20240502 01:59:36.413360 10131 raft_consensus.cc:729] T 1851ea6467584b0ea32a1f68696803d0 P 4b7f7ba2ac084e829c4304c16a94894f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4b7f7ba2ac084e829c4304c16a94894f, State: Running, Role: LEADER
I20240502 01:59:36.413350 10200 raft_consensus.cc:2454] T 1851ea6467584b0ea32a1f68696803d0 P e77d1c0d41034d1eb5a036dc17c071f5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdbecaeb8f214529b2012a9c003de2ff in term 2.
I20240502 01:59:36.413625 10131 consensus_queue.cc:260] T 1851ea6467584b0ea32a1f68696803d0 P 4b7f7ba2ac084e829c4304c16a94894f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "d6ff17402670432da6eeb5da2af7b4b6" member_type: NON_VOTER last_known_addr { host: "127.6.117.196" port: 34337 } attrs { promote: true } }
I20240502 01:59:36.413770 10021 leader_election.cc:302] T 1851ea6467584b0ea32a1f68696803d0 P bdbecaeb8f214529b2012a9c003de2ff [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bdbecaeb8f214529b2012a9c003de2ff, e77d1c0d41034d1eb5a036dc17c071f5; no voters:
I20240502 01:59:36.413847 10131 raft_consensus.cc:3037] T 1851ea6467584b0ea32a1f68696803d0 P 4b7f7ba2ac084e829c4304c16a94894f [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:59:36.413828 10706 consensus_queue.cc:237] T 580db11a61cb46f48d5f7507cfbdbb52 P bdbecaeb8f214529b2012a9c003de2ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } attrs { replace: true } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "d52d70d7578f4995b03ac6faba3ec739" member_type: VOTER last_known_addr { host: "127.6.117.197" port: 44467 } attrs { promote: false } }
I20240502 01:59:36.414152 10736 raft_consensus.cc:2781] T 1851ea6467584b0ea32a1f68696803d0 P bdbecaeb8f214529b2012a9c003de2ff [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:59:36.414227 10736 raft_consensus.cc:686] T 1851ea6467584b0ea32a1f68696803d0 P bdbecaeb8f214529b2012a9c003de2ff [term 2 LEADER]: Becoming Leader. State: Replica: bdbecaeb8f214529b2012a9c003de2ff, State: Running, Role: LEADER
I20240502 01:59:36.414453 10736 consensus_queue.cc:237] T 1851ea6467584b0ea32a1f68696803d0 P bdbecaeb8f214529b2012a9c003de2ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "d6ff17402670432da6eeb5da2af7b4b6" member_type: NON_VOTER last_known_addr { host: "127.6.117.196" port: 34337 } attrs { promote: true } }
I20240502 01:59:36.415102 10131 raft_consensus.cc:2454] T 1851ea6467584b0ea32a1f68696803d0 P 4b7f7ba2ac084e829c4304c16a94894f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdbecaeb8f214529b2012a9c003de2ff in term 2.
I20240502 01:59:36.416036 10131 raft_consensus.cc:1261] T 580db11a61cb46f48d5f7507cfbdbb52 P 4b7f7ba2ac084e829c4304c16a94894f [term 1 FOLLOWER]: Refusing update from remote peer bdbecaeb8f214529b2012a9c003de2ff: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240502 01:59:36.416038 10517 raft_consensus.cc:1261] T 580db11a61cb46f48d5f7507cfbdbb52 P d52d70d7578f4995b03ac6faba3ec739 [term 1 LEARNER]: Refusing update from remote peer bdbecaeb8f214529b2012a9c003de2ff: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240502 01:59:36.416482 10200 raft_consensus.cc:1261] T 580db11a61cb46f48d5f7507cfbdbb52 P e77d1c0d41034d1eb5a036dc17c071f5 [term 1 FOLLOWER]: Refusing update from remote peer bdbecaeb8f214529b2012a9c003de2ff: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240502 01:59:36.416379 9952 catalog_manager.cc:5484] T 1851ea6467584b0ea32a1f68696803d0 P bdbecaeb8f214529b2012a9c003de2ff reported cstate change: term changed from 1 to 2, leader changed from 4b7f7ba2ac084e829c4304c16a94894f (127.6.117.194) to bdbecaeb8f214529b2012a9c003de2ff (127.6.117.193). New cstate: current_term: 2 leader_uuid: "bdbecaeb8f214529b2012a9c003de2ff" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6ff17402670432da6eeb5da2af7b4b6" member_type: NON_VOTER last_known_addr { host: "127.6.117.196" port: 34337 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:59:36.416846 10741 consensus_queue.cc:1035] T 580db11a61cb46f48d5f7507cfbdbb52 P bdbecaeb8f214529b2012a9c003de2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "d52d70d7578f4995b03ac6faba3ec739" member_type: VOTER last_known_addr { host: "127.6.117.197" port: 44467 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 01:59:36.417073 10740 consensus_queue.cc:1035] T 580db11a61cb46f48d5f7507cfbdbb52 P bdbecaeb8f214529b2012a9c003de2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 01:59:36.417196 10737 consensus_queue.cc:1035] T 580db11a61cb46f48d5f7507cfbdbb52 P bdbecaeb8f214529b2012a9c003de2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 01:59:36.419900 10739 raft_consensus.cc:2932] T 580db11a61cb46f48d5f7507cfbdbb52 P bdbecaeb8f214529b2012a9c003de2ff [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index 26 to 27, d52d70d7578f4995b03ac6faba3ec739 (127.6.117.197) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } attrs { replace: true } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "d52d70d7578f4995b03ac6faba3ec739" member_type: VOTER last_known_addr { host: "127.6.117.197" port: 44467 } attrs { promote: false } } }
I20240502 01:59:36.420151 10517 raft_consensus.cc:2932] T 580db11a61cb46f48d5f7507cfbdbb52 P d52d70d7578f4995b03ac6faba3ec739 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, d52d70d7578f4995b03ac6faba3ec739 (127.6.117.197) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } attrs { replace: true } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "d52d70d7578f4995b03ac6faba3ec739" member_type: VOTER last_known_addr { host: "127.6.117.197" port: 44467 } attrs { promote: false } } }
I20240502 01:59:36.421947 10200 raft_consensus.cc:2932] T 580db11a61cb46f48d5f7507cfbdbb52 P e77d1c0d41034d1eb5a036dc17c071f5 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, d52d70d7578f4995b03ac6faba3ec739 (127.6.117.197) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } attrs { replace: true } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "d52d70d7578f4995b03ac6faba3ec739" member_type: VOTER last_known_addr { host: "127.6.117.197" port: 44467 } attrs { promote: false } } }
I20240502 01:59:36.422435 9952 catalog_manager.cc:5484] T 580db11a61cb46f48d5f7507cfbdbb52 P d52d70d7578f4995b03ac6faba3ec739 reported cstate change: config changed from index 26 to 27, d52d70d7578f4995b03ac6faba3ec739 (127.6.117.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bdbecaeb8f214529b2012a9c003de2ff" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } attrs { replace: true } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "d52d70d7578f4995b03ac6faba3ec739" member_type: VOTER last_known_addr { host: "127.6.117.197" port: 44467 } attrs { promote: false } } }
I20240502 01:59:36.421949 10129 raft_consensus.cc:2932] T 580db11a61cb46f48d5f7507cfbdbb52 P 4b7f7ba2ac084e829c4304c16a94894f [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, d52d70d7578f4995b03ac6faba3ec739 (127.6.117.197) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } attrs { replace: true } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "d52d70d7578f4995b03ac6faba3ec739" member_type: VOTER last_known_addr { host: "127.6.117.197" port: 44467 } attrs { promote: false } } }
I20240502 01:59:36.426106 10054 consensus_queue.cc:237] T 580db11a61cb46f48d5f7507cfbdbb52 P bdbecaeb8f214529b2012a9c003de2ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "d52d70d7578f4995b03ac6faba3ec739" member_type: VOTER last_known_addr { host: "127.6.117.197" port: 44467 } attrs { promote: false } }
I20240502 01:59:36.427249 10200 raft_consensus.cc:1261] T 580db11a61cb46f48d5f7507cfbdbb52 P e77d1c0d41034d1eb5a036dc17c071f5 [term 1 FOLLOWER]: Refusing update from remote peer bdbecaeb8f214529b2012a9c003de2ff: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240502 01:59:36.427542 10517 raft_consensus.cc:1261] T 580db11a61cb46f48d5f7507cfbdbb52 P d52d70d7578f4995b03ac6faba3ec739 [term 1 FOLLOWER]: Refusing update from remote peer bdbecaeb8f214529b2012a9c003de2ff: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240502 01:59:36.428196 10741 consensus_queue.cc:1035] T 580db11a61cb46f48d5f7507cfbdbb52 P bdbecaeb8f214529b2012a9c003de2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 01:59:36.428421 10706 consensus_queue.cc:1035] T 580db11a61cb46f48d5f7507cfbdbb52 P bdbecaeb8f214529b2012a9c003de2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "d52d70d7578f4995b03ac6faba3ec739" member_type: VOTER last_known_addr { host: "127.6.117.197" port: 44467 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 01:59:36.429761 10741 raft_consensus.cc:2932] T 580db11a61cb46f48d5f7507cfbdbb52 P bdbecaeb8f214529b2012a9c003de2ff [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER 4b7f7ba2ac084e829c4304c16a94894f (127.6.117.194) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "d52d70d7578f4995b03ac6faba3ec739" member_type: VOTER last_known_addr { host: "127.6.117.197" port: 44467 } attrs { promote: false } } }
I20240502 01:59:36.430176 10517 raft_consensus.cc:2932] T 580db11a61cb46f48d5f7507cfbdbb52 P d52d70d7578f4995b03ac6faba3ec739 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER 4b7f7ba2ac084e829c4304c16a94894f (127.6.117.194) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "d52d70d7578f4995b03ac6faba3ec739" member_type: VOTER last_known_addr { host: "127.6.117.197" port: 44467 } attrs { promote: false } } }
I20240502 01:59:36.430581 10200 raft_consensus.cc:2932] T 580db11a61cb46f48d5f7507cfbdbb52 P e77d1c0d41034d1eb5a036dc17c071f5 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER 4b7f7ba2ac084e829c4304c16a94894f (127.6.117.194) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "d52d70d7578f4995b03ac6faba3ec739" member_type: VOTER last_known_addr { host: "127.6.117.197" port: 44467 } attrs { promote: false } } }
I20240502 01:59:36.432245 9941 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 580db11a61cb46f48d5f7507cfbdbb52 with cas_config_opid_index 27: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:59:36.432684 9947 catalog_manager.cc:5484] T 580db11a61cb46f48d5f7507cfbdbb52 P e77d1c0d41034d1eb5a036dc17c071f5 reported cstate change: config changed from index 27 to 28, VOTER 4b7f7ba2ac084e829c4304c16a94894f (127.6.117.194) evicted. New cstate: current_term: 1 leader_uuid: "bdbecaeb8f214529b2012a9c003de2ff" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "d52d70d7578f4995b03ac6faba3ec739" member_type: VOTER last_known_addr { host: "127.6.117.197" port: 44467 } attrs { promote: false } } }
I20240502 01:59:36.434151 10200 raft_consensus.cc:1261] T e7a8fc5afcab4b1db4ebf245b66df4be P e77d1c0d41034d1eb5a036dc17c071f5 [term 4 FOLLOWER]: Refusing update from remote peer ae04d48c97204162b0f8a27d81fcec1a: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 4 index: 31. (index mismatch)
I20240502 01:59:36.434608 10717 consensus_queue.cc:1035] T e7a8fc5afcab4b1db4ebf245b66df4be P ae04d48c97204162b0f8a27d81fcec1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 01:59:36.435045 10116 tablet_service.cc:1514] Processing DeleteTablet for tablet 580db11a61cb46f48d5f7507cfbdbb52 with delete_type TABLET_DATA_TOMBSTONED (TS 4b7f7ba2ac084e829c4304c16a94894f not found in new config with opid_index 28) from {username='slave'} at 127.0.0.1:52598
I20240502 01:59:36.435417 10746 tablet_replica.cc:330] T 580db11a61cb46f48d5f7507cfbdbb52 P 4b7f7ba2ac084e829c4304c16a94894f: stopping tablet replica
I20240502 01:59:36.435704 10746 raft_consensus.cc:2229] T 580db11a61cb46f48d5f7507cfbdbb52 P 4b7f7ba2ac084e829c4304c16a94894f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:36.436033 10746 raft_consensus.cc:2258] T 580db11a61cb46f48d5f7507cfbdbb52 P 4b7f7ba2ac084e829c4304c16a94894f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:36.436641 10053 raft_consensus.cc:1261] T e7a8fc5afcab4b1db4ebf245b66df4be P bdbecaeb8f214529b2012a9c003de2ff [term 4 FOLLOWER]: Refusing update from remote peer ae04d48c97204162b0f8a27d81fcec1a: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 4 index: 31. (index mismatch)
I20240502 01:59:36.437044 10710 consensus_queue.cc:1035] T e7a8fc5afcab4b1db4ebf245b66df4be P ae04d48c97204162b0f8a27d81fcec1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 01:59:36.438787 10746 ts_tablet_manager.cc:1875] T 580db11a61cb46f48d5f7507cfbdbb52 P 4b7f7ba2ac084e829c4304c16a94894f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:59:36.441955 10746 ts_tablet_manager.cc:1888] T 580db11a61cb46f48d5f7507cfbdbb52 P 4b7f7ba2ac084e829c4304c16a94894f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.27
I20240502 01:59:36.442034 10746 log.cc:1196] T 580db11a61cb46f48d5f7507cfbdbb52 P 4b7f7ba2ac084e829c4304c16a94894f: Deleting WAL directory at /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714615076189642-6615-0/minicluster-data/ts-1-root/wals/580db11a61cb46f48d5f7507cfbdbb52
I20240502 01:59:36.442514 9940 catalog_manager.cc:4830] TS 4b7f7ba2ac084e829c4304c16a94894f (127.6.117.194:46805): tablet 580db11a61cb46f48d5f7507cfbdbb52 (table test-workload [id=10a74377088c4a9890835737184010db]) successfully deleted
I20240502 01:59:36.646654 10731 raft_consensus.cc:1050] T 19ff9c216a094fde86ea9f9648bf08cb P 4b7f7ba2ac084e829c4304c16a94894f: attempting to promote NON_VOTER d52d70d7578f4995b03ac6faba3ec739 to VOTER
I20240502 01:59:36.647068 10731 consensus_queue.cc:237] T 19ff9c216a094fde86ea9f9648bf08cb P 4b7f7ba2ac084e829c4304c16a94894f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } attrs { replace: true } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "d52d70d7578f4995b03ac6faba3ec739" member_type: VOTER last_known_addr { host: "127.6.117.197" port: 44467 } attrs { promote: false } }
I20240502 01:59:36.648216 10200 raft_consensus.cc:1261] T 19ff9c216a094fde86ea9f9648bf08cb P e77d1c0d41034d1eb5a036dc17c071f5 [term 1 FOLLOWER]: Refusing update from remote peer 4b7f7ba2ac084e829c4304c16a94894f: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240502 01:59:36.648308 10053 raft_consensus.cc:1261] T 19ff9c216a094fde86ea9f9648bf08cb P bdbecaeb8f214529b2012a9c003de2ff [term 1 FOLLOWER]: Refusing update from remote peer 4b7f7ba2ac084e829c4304c16a94894f: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240502 01:59:36.648483 10517 raft_consensus.cc:1261] T 19ff9c216a094fde86ea9f9648bf08cb P d52d70d7578f4995b03ac6faba3ec739 [term 1 LEARNER]: Refusing update from remote peer 4b7f7ba2ac084e829c4304c16a94894f: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240502 01:59:36.648802 10729 consensus_queue.cc:1035] T 19ff9c216a094fde86ea9f9648bf08cb P 4b7f7ba2ac084e829c4304c16a94894f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 01:59:36.648990 10728 consensus_queue.cc:1035] T 19ff9c216a094fde86ea9f9648bf08cb P 4b7f7ba2ac084e829c4304c16a94894f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 01:59:36.649174 10677 consensus_queue.cc:1035] T 19ff9c216a094fde86ea9f9648bf08cb P 4b7f7ba2ac084e829c4304c16a94894f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d52d70d7578f4995b03ac6faba3ec739" member_type: VOTER last_known_addr { host: "127.6.117.197" port: 44467 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 01:59:36.651119 10677 raft_consensus.cc:2932] T 19ff9c216a094fde86ea9f9648bf08cb P 4b7f7ba2ac084e829c4304c16a94894f [term 1 LEADER]: Committing config change with OpId 1.26: config changed from index 25 to 26, d52d70d7578f4995b03ac6faba3ec739 (127.6.117.197) changed from NON_VOTER to VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } attrs { replace: true } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "d52d70d7578f4995b03ac6faba3ec739" member_type: VOTER last_known_addr { host: "127.6.117.197" port: 44467 } attrs { promote: false } } }
I20240502 01:59:36.651470 10517 raft_consensus.cc:2932] T 19ff9c216a094fde86ea9f9648bf08cb P d52d70d7578f4995b03ac6faba3ec739 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index 25 to 26, d52d70d7578f4995b03ac6faba3ec739 (127.6.117.197) changed from NON_VOTER to VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } attrs { replace: true } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "d52d70d7578f4995b03ac6faba3ec739" member_type: VOTER last_known_addr { host: "127.6.117.197" port: 44467 } attrs { promote: false } } }
I20240502 01:59:36.651592 10202 raft_consensus.cc:2932] T 19ff9c216a094fde86ea9f9648bf08cb P e77d1c0d41034d1eb5a036dc17c071f5 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index 25 to 26, d52d70d7578f4995b03ac6faba3ec739 (127.6.117.197) changed from NON_VOTER to VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } attrs { replace: true } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "d52d70d7578f4995b03ac6faba3ec739" member_type: VOTER last_known_addr { host: "127.6.117.197" port: 44467 } attrs { promote: false } } }
I20240502 01:59:36.653632 10054 raft_consensus.cc:2932] T 19ff9c216a094fde86ea9f9648bf08cb P bdbecaeb8f214529b2012a9c003de2ff [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index 25 to 26, d52d70d7578f4995b03ac6faba3ec739 (127.6.117.197) changed from NON_VOTER to VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } attrs { replace: true } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "d52d70d7578f4995b03ac6faba3ec739" member_type: VOTER last_known_addr { host: "127.6.117.197" port: 44467 } attrs { promote: false } } }
I20240502 01:59:36.653754 9953 catalog_manager.cc:5484] T 19ff9c216a094fde86ea9f9648bf08cb P e77d1c0d41034d1eb5a036dc17c071f5 reported cstate change: config changed from index 25 to 26, d52d70d7578f4995b03ac6faba3ec739 (127.6.117.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4b7f7ba2ac084e829c4304c16a94894f" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } attrs { replace: true } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "d52d70d7578f4995b03ac6faba3ec739" member_type: VOTER last_known_addr { host: "127.6.117.197" port: 44467 } attrs { promote: false } } }
I20240502 01:59:36.656536 10131 consensus_queue.cc:237] T 19ff9c216a094fde86ea9f9648bf08cb P 4b7f7ba2ac084e829c4304c16a94894f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "d52d70d7578f4995b03ac6faba3ec739" member_type: VOTER last_known_addr { host: "127.6.117.197" port: 44467 } attrs { promote: false } }
I20240502 01:59:36.657534 10517 raft_consensus.cc:1261] T 19ff9c216a094fde86ea9f9648bf08cb P d52d70d7578f4995b03ac6faba3ec739 [term 1 FOLLOWER]: Refusing update from remote peer 4b7f7ba2ac084e829c4304c16a94894f: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240502 01:59:36.657585 10200 raft_consensus.cc:1261] T 19ff9c216a094fde86ea9f9648bf08cb P e77d1c0d41034d1eb5a036dc17c071f5 [term 1 FOLLOWER]: Refusing update from remote peer 4b7f7ba2ac084e829c4304c16a94894f: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240502 01:59:36.657896 10677 consensus_queue.cc:1035] T 19ff9c216a094fde86ea9f9648bf08cb P 4b7f7ba2ac084e829c4304c16a94894f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d52d70d7578f4995b03ac6faba3ec739" member_type: VOTER last_known_addr { host: "127.6.117.197" port: 44467 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 01:59:36.658069 10731 consensus_queue.cc:1035] T 19ff9c216a094fde86ea9f9648bf08cb P 4b7f7ba2ac084e829c4304c16a94894f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 01:59:36.659494 10729 raft_consensus.cc:2932] T 19ff9c216a094fde86ea9f9648bf08cb P 4b7f7ba2ac084e829c4304c16a94894f [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index 26 to 27, VOTER bdbecaeb8f214529b2012a9c003de2ff (127.6.117.193) evicted. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "d52d70d7578f4995b03ac6faba3ec739" member_type: VOTER last_known_addr { host: "127.6.117.197" port: 44467 } attrs { promote: false } } }
I20240502 01:59:36.659911 10517 raft_consensus.cc:2932] T 19ff9c216a094fde86ea9f9648bf08cb P d52d70d7578f4995b03ac6faba3ec739 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, VOTER bdbecaeb8f214529b2012a9c003de2ff (127.6.117.193) evicted. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "d52d70d7578f4995b03ac6faba3ec739" member_type: VOTER last_known_addr { host: "127.6.117.197" port: 44467 } attrs { promote: false } } }
I20240502 01:59:36.660432 10200 raft_consensus.cc:2932] T 19ff9c216a094fde86ea9f9648bf08cb P e77d1c0d41034d1eb5a036dc17c071f5 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, VOTER bdbecaeb8f214529b2012a9c003de2ff (127.6.117.193) evicted. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "d52d70d7578f4995b03ac6faba3ec739" member_type: VOTER last_known_addr { host: "127.6.117.197" port: 44467 } attrs { promote: false } } }
I20240502 01:59:36.661188 9940 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 19ff9c216a094fde86ea9f9648bf08cb with cas_config_opid_index 26: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:59:36.662063 9953 catalog_manager.cc:5484] T 19ff9c216a094fde86ea9f9648bf08cb P 4b7f7ba2ac084e829c4304c16a94894f reported cstate change: config changed from index 26 to 27, VOTER bdbecaeb8f214529b2012a9c003de2ff (127.6.117.193) evicted. New cstate: current_term: 1 leader_uuid: "4b7f7ba2ac084e829c4304c16a94894f" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d52d70d7578f4995b03ac6faba3ec739" member_type: VOTER last_known_addr { host: "127.6.117.197" port: 44467 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:59:36.664873 10044 tablet_service.cc:1514] Processing DeleteTablet for tablet 19ff9c216a094fde86ea9f9648bf08cb with delete_type TABLET_DATA_TOMBSTONED (TS bdbecaeb8f214529b2012a9c003de2ff not found in new config with opid_index 27) from {username='slave'} at 127.0.0.1:44390
I20240502 01:59:36.665246 10754 tablet_replica.cc:330] T 19ff9c216a094fde86ea9f9648bf08cb P bdbecaeb8f214529b2012a9c003de2ff: stopping tablet replica
I20240502 01:59:36.665362 10754 raft_consensus.cc:2229] T 19ff9c216a094fde86ea9f9648bf08cb P bdbecaeb8f214529b2012a9c003de2ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:36.665549 10754 raft_consensus.cc:2258] T 19ff9c216a094fde86ea9f9648bf08cb P bdbecaeb8f214529b2012a9c003de2ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:36.666091 10754 ts_tablet_manager.cc:1875] T 19ff9c216a094fde86ea9f9648bf08cb P bdbecaeb8f214529b2012a9c003de2ff: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:59:36.667987 10754 ts_tablet_manager.cc:1888] T 19ff9c216a094fde86ea9f9648bf08cb P bdbecaeb8f214529b2012a9c003de2ff: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.26
I20240502 01:59:36.668063 10754 log.cc:1196] T 19ff9c216a094fde86ea9f9648bf08cb P bdbecaeb8f214529b2012a9c003de2ff: Deleting WAL directory at /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714615076189642-6615-0/minicluster-data/ts-0-root/wals/19ff9c216a094fde86ea9f9648bf08cb
I20240502 01:59:36.668543 9941 catalog_manager.cc:4830] TS bdbecaeb8f214529b2012a9c003de2ff (127.6.117.193:34597): tablet 19ff9c216a094fde86ea9f9648bf08cb (table test-workload [id=10a74377088c4a9890835737184010db]) successfully deleted
I20240502 01:59:36.819938 10441 raft_consensus.cc:3037] T 1851ea6467584b0ea32a1f68696803d0 P d6ff17402670432da6eeb5da2af7b4b6 [term 1 LEARNER]: Advancing to term 2
I20240502 01:59:36.821053 10441 raft_consensus.cc:1261] T 1851ea6467584b0ea32a1f68696803d0 P d6ff17402670432da6eeb5da2af7b4b6 [term 2 LEARNER]: Refusing update from remote peer bdbecaeb8f214529b2012a9c003de2ff: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240502 01:59:36.821382 10706 consensus_queue.cc:1035] T 1851ea6467584b0ea32a1f68696803d0 P bdbecaeb8f214529b2012a9c003de2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6ff17402670432da6eeb5da2af7b4b6" member_type: NON_VOTER last_known_addr { host: "127.6.117.196" port: 34337 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 01:59:36.823076 10706 raft_consensus.cc:1050] T 1851ea6467584b0ea32a1f68696803d0 P bdbecaeb8f214529b2012a9c003de2ff: attempting to promote NON_VOTER d6ff17402670432da6eeb5da2af7b4b6 to VOTER
W20240502 01:59:36.823158 10706 raft_consensus.cc:1053] T 1851ea6467584b0ea32a1f68696803d0 P bdbecaeb8f214529b2012a9c003de2ff: Unable to promote non-voter d6ff17402670432da6eeb5da2af7b4b6: Illegal state: Leader has not yet committed an operation in its own term
I20240502 01:59:36.847592 10131 raft_consensus.cc:1261] T 1851ea6467584b0ea32a1f68696803d0 P 4b7f7ba2ac084e829c4304c16a94894f [term 2 FOLLOWER]: Refusing update from remote peer bdbecaeb8f214529b2012a9c003de2ff: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240502 01:59:36.847923 10706 consensus_queue.cc:1035] T 1851ea6467584b0ea32a1f68696803d0 P bdbecaeb8f214529b2012a9c003de2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 01:59:36.850414 10200 raft_consensus.cc:1261] T 1851ea6467584b0ea32a1f68696803d0 P e77d1c0d41034d1eb5a036dc17c071f5 [term 2 FOLLOWER]: Refusing update from remote peer bdbecaeb8f214529b2012a9c003de2ff: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240502 01:59:36.851382 10736 consensus_queue.cc:1035] T 1851ea6467584b0ea32a1f68696803d0 P bdbecaeb8f214529b2012a9c003de2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 01:59:36.851766 10740 raft_consensus.cc:1050] T 1851ea6467584b0ea32a1f68696803d0 P bdbecaeb8f214529b2012a9c003de2ff: attempting to promote NON_VOTER d6ff17402670432da6eeb5da2af7b4b6 to VOTER
I20240502 01:59:36.852119 10740 consensus_queue.cc:237] T 1851ea6467584b0ea32a1f68696803d0 P bdbecaeb8f214529b2012a9c003de2ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 26, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "d6ff17402670432da6eeb5da2af7b4b6" member_type: VOTER last_known_addr { host: "127.6.117.196" port: 34337 } attrs { promote: false } }
I20240502 01:59:36.853291 10441 raft_consensus.cc:1261] T 1851ea6467584b0ea32a1f68696803d0 P d6ff17402670432da6eeb5da2af7b4b6 [term 2 LEARNER]: Refusing update from remote peer bdbecaeb8f214529b2012a9c003de2ff: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240502 01:59:36.854107 10131 raft_consensus.cc:1261] T 1851ea6467584b0ea32a1f68696803d0 P 4b7f7ba2ac084e829c4304c16a94894f [term 2 FOLLOWER]: Refusing update from remote peer bdbecaeb8f214529b2012a9c003de2ff: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240502 01:59:36.854092 10706 consensus_queue.cc:1035] T 1851ea6467584b0ea32a1f68696803d0 P bdbecaeb8f214529b2012a9c003de2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6ff17402670432da6eeb5da2af7b4b6" member_type: VOTER last_known_addr { host: "127.6.117.196" port: 34337 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 01:59:36.855480 10202 raft_consensus.cc:1261] T 1851ea6467584b0ea32a1f68696803d0 P e77d1c0d41034d1eb5a036dc17c071f5 [term 2 FOLLOWER]: Refusing update from remote peer bdbecaeb8f214529b2012a9c003de2ff: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240502 01:59:36.855656 10740 consensus_queue.cc:1035] T 1851ea6467584b0ea32a1f68696803d0 P bdbecaeb8f214529b2012a9c003de2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 01:59:36.855923 10736 consensus_queue.cc:1035] T 1851ea6467584b0ea32a1f68696803d0 P bdbecaeb8f214529b2012a9c003de2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 01:59:36.857197 10740 raft_consensus.cc:2932] T 1851ea6467584b0ea32a1f68696803d0 P bdbecaeb8f214529b2012a9c003de2ff [term 2 LEADER]: Committing config change with OpId 2.28: config changed from index 26 to 28, d6ff17402670432da6eeb5da2af7b4b6 (127.6.117.196) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "d6ff17402670432da6eeb5da2af7b4b6" member_type: VOTER last_known_addr { host: "127.6.117.196" port: 34337 } attrs { promote: false } } }
I20240502 01:59:36.857707 10131 raft_consensus.cc:2932] T 1851ea6467584b0ea32a1f68696803d0 P 4b7f7ba2ac084e829c4304c16a94894f [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index 26 to 28, d6ff17402670432da6eeb5da2af7b4b6 (127.6.117.196) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "d6ff17402670432da6eeb5da2af7b4b6" member_type: VOTER last_known_addr { host: "127.6.117.196" port: 34337 } attrs { promote: false } } }
I20240502 01:59:36.859314 10441 raft_consensus.cc:2932] T 1851ea6467584b0ea32a1f68696803d0 P d6ff17402670432da6eeb5da2af7b4b6 [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index 26 to 28, d6ff17402670432da6eeb5da2af7b4b6 (127.6.117.196) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "d6ff17402670432da6eeb5da2af7b4b6" member_type: VOTER last_known_addr { host: "127.6.117.196" port: 34337 } attrs { promote: false } } }
I20240502 01:59:36.859333 10200 raft_consensus.cc:2932] T 1851ea6467584b0ea32a1f68696803d0 P e77d1c0d41034d1eb5a036dc17c071f5 [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index 26 to 28, d6ff17402670432da6eeb5da2af7b4b6 (127.6.117.196) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "d6ff17402670432da6eeb5da2af7b4b6" member_type: VOTER last_known_addr { host: "127.6.117.196" port: 34337 } attrs { promote: false } } }
I20240502 01:59:36.859789 9952 catalog_manager.cc:5484] T 1851ea6467584b0ea32a1f68696803d0 P 4b7f7ba2ac084e829c4304c16a94894f reported cstate change: config changed from index 26 to 28, d6ff17402670432da6eeb5da2af7b4b6 (127.6.117.196) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "bdbecaeb8f214529b2012a9c003de2ff" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "d6ff17402670432da6eeb5da2af7b4b6" member_type: VOTER last_known_addr { host: "127.6.117.196" port: 34337 } attrs { promote: false } } }
I20240502 01:59:36.863230 10054 consensus_queue.cc:237] T 1851ea6467584b0ea32a1f68696803d0 P bdbecaeb8f214529b2012a9c003de2ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 26, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "d6ff17402670432da6eeb5da2af7b4b6" member_type: VOTER last_known_addr { host: "127.6.117.196" port: 34337 } attrs { promote: false } }
I20240502 01:59:36.864216 10441 raft_consensus.cc:1261] T 1851ea6467584b0ea32a1f68696803d0 P d6ff17402670432da6eeb5da2af7b4b6 [term 2 FOLLOWER]: Refusing update from remote peer bdbecaeb8f214529b2012a9c003de2ff: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240502 01:59:36.864297 10131 raft_consensus.cc:1261] T 1851ea6467584b0ea32a1f68696803d0 P 4b7f7ba2ac084e829c4304c16a94894f [term 2 FOLLOWER]: Refusing update from remote peer bdbecaeb8f214529b2012a9c003de2ff: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240502 01:59:36.864593 10739 consensus_queue.cc:1035] T 1851ea6467584b0ea32a1f68696803d0 P bdbecaeb8f214529b2012a9c003de2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6ff17402670432da6eeb5da2af7b4b6" member_type: VOTER last_known_addr { host: "127.6.117.196" port: 34337 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 01:59:36.864779 10737 consensus_queue.cc:1035] T 1851ea6467584b0ea32a1f68696803d0 P bdbecaeb8f214529b2012a9c003de2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 01:59:36.866122 10741 raft_consensus.cc:2932] T 1851ea6467584b0ea32a1f68696803d0 P bdbecaeb8f214529b2012a9c003de2ff [term 2 LEADER]: Committing config change with OpId 2.29: config changed from index 28 to 29, VOTER e77d1c0d41034d1eb5a036dc17c071f5 (127.6.117.195) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "d6ff17402670432da6eeb5da2af7b4b6" member_type: VOTER last_known_addr { host: "127.6.117.196" port: 34337 } attrs { promote: false } } }
I20240502 01:59:36.866534 10441 raft_consensus.cc:2932] T 1851ea6467584b0ea32a1f68696803d0 P d6ff17402670432da6eeb5da2af7b4b6 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 28 to 29, VOTER e77d1c0d41034d1eb5a036dc17c071f5 (127.6.117.195) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "d6ff17402670432da6eeb5da2af7b4b6" member_type: VOTER last_known_addr { host: "127.6.117.196" port: 34337 } attrs { promote: false } } }
I20240502 01:59:36.867810 9941 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1851ea6467584b0ea32a1f68696803d0 with cas_config_opid_index 28: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:59:36.868077 10131 raft_consensus.cc:2932] T 1851ea6467584b0ea32a1f68696803d0 P 4b7f7ba2ac084e829c4304c16a94894f [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 28 to 29, VOTER e77d1c0d41034d1eb5a036dc17c071f5 (127.6.117.195) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "d6ff17402670432da6eeb5da2af7b4b6" member_type: VOTER last_known_addr { host: "127.6.117.196" port: 34337 } attrs { promote: false } } }
I20240502 01:59:36.868302 9947 catalog_manager.cc:5484] T 1851ea6467584b0ea32a1f68696803d0 P d6ff17402670432da6eeb5da2af7b4b6 reported cstate change: config changed from index 28 to 29, VOTER e77d1c0d41034d1eb5a036dc17c071f5 (127.6.117.195) evicted. New cstate: current_term: 2 leader_uuid: "bdbecaeb8f214529b2012a9c003de2ff" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "d6ff17402670432da6eeb5da2af7b4b6" member_type: VOTER last_known_addr { host: "127.6.117.196" port: 34337 } attrs { promote: false } } }
I20240502 01:59:36.870839 10192 tablet_service.cc:1514] Processing DeleteTablet for tablet 1851ea6467584b0ea32a1f68696803d0 with delete_type TABLET_DATA_TOMBSTONED (TS e77d1c0d41034d1eb5a036dc17c071f5 not found in new config with opid_index 29) from {username='slave'} at 127.0.0.1:48060
I20240502 01:59:36.871222 10763 tablet_replica.cc:330] T 1851ea6467584b0ea32a1f68696803d0 P e77d1c0d41034d1eb5a036dc17c071f5: stopping tablet replica
I20240502 01:59:36.871361 10763 raft_consensus.cc:2229] T 1851ea6467584b0ea32a1f68696803d0 P e77d1c0d41034d1eb5a036dc17c071f5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:36.871544 10763 raft_consensus.cc:2258] T 1851ea6467584b0ea32a1f68696803d0 P e77d1c0d41034d1eb5a036dc17c071f5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:36.872154 10763 ts_tablet_manager.cc:1875] T 1851ea6467584b0ea32a1f68696803d0 P e77d1c0d41034d1eb5a036dc17c071f5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:59:36.873948 10763 ts_tablet_manager.cc:1888] T 1851ea6467584b0ea32a1f68696803d0 P e77d1c0d41034d1eb5a036dc17c071f5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.28
I20240502 01:59:36.874018 10763 log.cc:1196] T 1851ea6467584b0ea32a1f68696803d0 P e77d1c0d41034d1eb5a036dc17c071f5: Deleting WAL directory at /tmp/dist-test-taskdXw3Ep/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714615076189642-6615-0/minicluster-data/ts-2-root/wals/1851ea6467584b0ea32a1f68696803d0
I20240502 01:59:36.874466 9939 catalog_manager.cc:4830] TS e77d1c0d41034d1eb5a036dc17c071f5 (127.6.117.195:43977): tablet 1851ea6467584b0ea32a1f68696803d0 (table test-workload [id=10a74377088c4a9890835737184010db]) successfully deleted
I20240502 01:59:37.881592 10590 consensus_queue.cc:237] T e7a8fc5afcab4b1db4ebf245b66df4be P ae04d48c97204162b0f8a27d81fcec1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 4.31, Last appended by leader: 30, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } attrs { replace: true } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "ae04d48c97204162b0f8a27d81fcec1a" member_type: VOTER last_known_addr { host: "127.6.117.198" port: 33655 } attrs { promote: false } } peers { permanent_uuid: "d6ff17402670432da6eeb5da2af7b4b6" member_type: NON_VOTER last_known_addr { host: "127.6.117.196" port: 34337 } attrs { promote: true } }
I20240502 01:59:37.883100 10053 raft_consensus.cc:1261] T e7a8fc5afcab4b1db4ebf245b66df4be P bdbecaeb8f214529b2012a9c003de2ff [term 4 FOLLOWER]: Refusing update from remote peer ae04d48c97204162b0f8a27d81fcec1a: Log matching property violated. Preceding OpId in replica: term: 4 index: 31. Preceding OpId from leader: term: 4 index: 32. (index mismatch)
I20240502 01:59:37.883445 10200 raft_consensus.cc:1261] T e7a8fc5afcab4b1db4ebf245b66df4be P e77d1c0d41034d1eb5a036dc17c071f5 [term 4 FOLLOWER]: Refusing update from remote peer ae04d48c97204162b0f8a27d81fcec1a: Log matching property violated. Preceding OpId in replica: term: 4 index: 31. Preceding OpId from leader: term: 4 index: 32. (index mismatch)
I20240502 01:59:37.883564 10766 consensus_queue.cc:1035] T e7a8fc5afcab4b1db4ebf245b66df4be P ae04d48c97204162b0f8a27d81fcec1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240502 01:59:37.883759 10770 consensus_queue.cc:1035] T e7a8fc5afcab4b1db4ebf245b66df4be P ae04d48c97204162b0f8a27d81fcec1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240502 01:59:37.885428 10766 raft_consensus.cc:2932] T e7a8fc5afcab4b1db4ebf245b66df4be P ae04d48c97204162b0f8a27d81fcec1a [term 4 LEADER]: Committing config change with OpId 4.32: config changed from index 30 to 32, NON_VOTER d6ff17402670432da6eeb5da2af7b4b6 (127.6.117.196) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } attrs { replace: true } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "ae04d48c97204162b0f8a27d81fcec1a" member_type: VOTER last_known_addr { host: "127.6.117.198" port: 33655 } attrs { promote: false } } peers { permanent_uuid: "d6ff17402670432da6eeb5da2af7b4b6" member_type: NON_VOTER last_known_addr { host: "127.6.117.196" port: 34337 } attrs { promote: true } } }
I20240502 01:59:37.885811 10200 raft_consensus.cc:2932] T e7a8fc5afcab4b1db4ebf245b66df4be P e77d1c0d41034d1eb5a036dc17c071f5 [term 4 FOLLOWER]: Committing config change with OpId 4.32: config changed from index 30 to 32, NON_VOTER d6ff17402670432da6eeb5da2af7b4b6 (127.6.117.196) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } attrs { replace: true } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "ae04d48c97204162b0f8a27d81fcec1a" member_type: VOTER last_known_addr { host: "127.6.117.198" port: 33655 } attrs { promote: false } } peers { permanent_uuid: "d6ff17402670432da6eeb5da2af7b4b6" member_type: NON_VOTER last_known_addr { host: "127.6.117.196" port: 34337 } attrs { promote: true } } }
I20240502 01:59:37.886318 10053 raft_consensus.cc:2932] T e7a8fc5afcab4b1db4ebf245b66df4be P bdbecaeb8f214529b2012a9c003de2ff [term 4 FOLLOWER]: Committing config change with OpId 4.32: config changed from index 30 to 32, NON_VOTER d6ff17402670432da6eeb5da2af7b4b6 (127.6.117.196) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } attrs { replace: true } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "ae04d48c97204162b0f8a27d81fcec1a" member_type: VOTER last_known_addr { host: "127.6.117.198" port: 33655 } attrs { promote: false } } peers { permanent_uuid: "d6ff17402670432da6eeb5da2af7b4b6" member_type: NON_VOTER last_known_addr { host: "127.6.117.196" port: 34337 } attrs { promote: true } } }
I20240502 01:59:37.888505 9947 catalog_manager.cc:5484] T e7a8fc5afcab4b1db4ebf245b66df4be P ae04d48c97204162b0f8a27d81fcec1a reported cstate change: config changed from index 30 to 32, NON_VOTER d6ff17402670432da6eeb5da2af7b4b6 (127.6.117.196) added. New cstate: current_term: 4 leader_uuid: "ae04d48c97204162b0f8a27d81fcec1a" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae04d48c97204162b0f8a27d81fcec1a" member_type: VOTER last_known_addr { host: "127.6.117.198" port: 33655 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6ff17402670432da6eeb5da2af7b4b6" member_type: NON_VOTER last_known_addr { host: "127.6.117.196" port: 34337 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:59:37.888592 10053 consensus_queue.cc:237] T abf7d8b411bb46b68ac9ae77d74f8e2b P bdbecaeb8f214529b2012a9c003de2ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "ae04d48c97204162b0f8a27d81fcec1a" member_type: NON_VOTER last_known_addr { host: "127.6.117.198" port: 33655 } attrs { promote: true } }
W20240502 01:59:37.889572 10556 consensus_peers.cc:485] T e7a8fc5afcab4b1db4ebf245b66df4be P ae04d48c97204162b0f8a27d81fcec1a -> Peer d6ff17402670432da6eeb5da2af7b4b6 (127.6.117.196:34337): Couldn't send request to peer d6ff17402670432da6eeb5da2af7b4b6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e7a8fc5afcab4b1db4ebf245b66df4be. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:37.890000 10131 raft_consensus.cc:1261] T abf7d8b411bb46b68ac9ae77d74f8e2b P 4b7f7ba2ac084e829c4304c16a94894f [term 1 FOLLOWER]: Refusing update from remote peer bdbecaeb8f214529b2012a9c003de2ff: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240502 01:59:37.890340 10739 consensus_queue.cc:1035] T abf7d8b411bb46b68ac9ae77d74f8e2b P bdbecaeb8f214529b2012a9c003de2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 01:59:37.890547 10200 raft_consensus.cc:1261] T abf7d8b411bb46b68ac9ae77d74f8e2b P e77d1c0d41034d1eb5a036dc17c071f5 [term 1 FOLLOWER]: Refusing update from remote peer bdbecaeb8f214529b2012a9c003de2ff: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
W20240502 01:59:37.890864 10021 consensus_peers.cc:485] T abf7d8b411bb46b68ac9ae77d74f8e2b P bdbecaeb8f214529b2012a9c003de2ff -> Peer ae04d48c97204162b0f8a27d81fcec1a (127.6.117.198:33655): Couldn't send request to peer ae04d48c97204162b0f8a27d81fcec1a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: abf7d8b411bb46b68ac9ae77d74f8e2b. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:37.891322 10739 consensus_queue.cc:1035] T abf7d8b411bb46b68ac9ae77d74f8e2b P bdbecaeb8f214529b2012a9c003de2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 01:59:37.892493 10782 raft_consensus.cc:2932] T abf7d8b411bb46b68ac9ae77d74f8e2b P bdbecaeb8f214529b2012a9c003de2ff [term 1 LEADER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER ae04d48c97204162b0f8a27d81fcec1a (127.6.117.198) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "ae04d48c97204162b0f8a27d81fcec1a" member_type: NON_VOTER last_known_addr { host: "127.6.117.198" port: 33655 } attrs { promote: true } } }
I20240502 01:59:37.892849 10131 raft_consensus.cc:2932] T abf7d8b411bb46b68ac9ae77d74f8e2b P 4b7f7ba2ac084e829c4304c16a94894f [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER ae04d48c97204162b0f8a27d81fcec1a (127.6.117.198) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "ae04d48c97204162b0f8a27d81fcec1a" member_type: NON_VOTER last_known_addr { host: "127.6.117.198" port: 33655 } attrs { promote: true } } }
I20240502 01:59:37.893685 10200 raft_consensus.cc:2932] T abf7d8b411bb46b68ac9ae77d74f8e2b P e77d1c0d41034d1eb5a036dc17c071f5 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER ae04d48c97204162b0f8a27d81fcec1a (127.6.117.198) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "ae04d48c97204162b0f8a27d81fcec1a" member_type: NON_VOTER last_known_addr { host: "127.6.117.198" port: 33655 } attrs { promote: true } } }
I20240502 01:59:37.895198 10053 consensus_queue.cc:237] T 1851ea6467584b0ea32a1f68696803d0 P bdbecaeb8f214529b2012a9c003de2ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 26, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } attrs { replace: true } } peers { permanent_uuid: "d6ff17402670432da6eeb5da2af7b4b6" member_type: VOTER last_known_addr { host: "127.6.117.196" port: 34337 } attrs { promote: false } } peers { permanent_uuid: "d52d70d7578f4995b03ac6faba3ec739" member_type: NON_VOTER last_known_addr { host: "127.6.117.197" port: 44467 } attrs { promote: true } }
I20240502 01:59:37.895723 9948 catalog_manager.cc:5484] T abf7d8b411bb46b68ac9ae77d74f8e2b P e77d1c0d41034d1eb5a036dc17c071f5 reported cstate change: config changed from index -1 to 26, NON_VOTER ae04d48c97204162b0f8a27d81fcec1a (127.6.117.198) added. New cstate: current_term: 1 leader_uuid: "bdbecaeb8f214529b2012a9c003de2ff" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "ae04d48c97204162b0f8a27d81fcec1a" member_type: NON_VOTER last_known_addr { host: "127.6.117.198" port: 33655 } attrs { promote: true } } }
I20240502 01:59:37.896625 10441 raft_consensus.cc:1261] T 1851ea6467584b0ea32a1f68696803d0 P d6ff17402670432da6eeb5da2af7b4b6 [term 2 FOLLOWER]: Refusing update from remote peer bdbecaeb8f214529b2012a9c003de2ff: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240502 01:59:37.896939 10131 raft_consensus.cc:1261] T 1851ea6467584b0ea32a1f68696803d0 P 4b7f7ba2ac084e829c4304c16a94894f [term 2 FOLLOWER]: Refusing update from remote peer bdbecaeb8f214529b2012a9c003de2ff: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240502 01:59:37.896955 10739 consensus_queue.cc:1035] T 1851ea6467584b0ea32a1f68696803d0 P bdbecaeb8f214529b2012a9c003de2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6ff17402670432da6eeb5da2af7b4b6" member_type: VOTER last_known_addr { host: "127.6.117.196" port: 34337 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 01:59:37.897394 10739 consensus_queue.cc:1035] T 1851ea6467584b0ea32a1f68696803d0 P bdbecaeb8f214529b2012a9c003de2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
W20240502 01:59:37.897814 10020 consensus_peers.cc:485] T 1851ea6467584b0ea32a1f68696803d0 P bdbecaeb8f214529b2012a9c003de2ff -> Peer d52d70d7578f4995b03ac6faba3ec739 (127.6.117.197:44467): Couldn't send request to peer d52d70d7578f4995b03ac6faba3ec739. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1851ea6467584b0ea32a1f68696803d0. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:37.898859 10739 raft_consensus.cc:2932] T 1851ea6467584b0ea32a1f68696803d0 P bdbecaeb8f214529b2012a9c003de2ff [term 2 LEADER]: Committing config change with OpId 2.30: config changed from index 29 to 30, NON_VOTER d52d70d7578f4995b03ac6faba3ec739 (127.6.117.197) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } attrs { replace: true } } peers { permanent_uuid: "d6ff17402670432da6eeb5da2af7b4b6" member_type: VOTER last_known_addr { host: "127.6.117.196" port: 34337 } attrs { promote: false } } peers { permanent_uuid: "d52d70d7578f4995b03ac6faba3ec739" member_type: NON_VOTER last_known_addr { host: "127.6.117.197" port: 44467 } attrs { promote: true } } }
I20240502 01:59:37.899246 10131 raft_consensus.cc:2932] T 1851ea6467584b0ea32a1f68696803d0 P 4b7f7ba2ac084e829c4304c16a94894f [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 29 to 30, NON_VOTER d52d70d7578f4995b03ac6faba3ec739 (127.6.117.197) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } attrs { replace: true } } peers { permanent_uuid: "d6ff17402670432da6eeb5da2af7b4b6" member_type: VOTER last_known_addr { host: "127.6.117.196" port: 34337 } attrs { promote: false } } peers { permanent_uuid: "d52d70d7578f4995b03ac6faba3ec739" member_type: NON_VOTER last_known_addr { host: "127.6.117.197" port: 44467 } attrs { promote: true } } }
I20240502 01:59:37.901108 10200 consensus_queue.cc:237] T 4885222f238c4d8ea20ebc4c7e171694 P e77d1c0d41034d1eb5a036dc17c071f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 2.26, Last appended by leader: 22, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } attrs { replace: true } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "d52d70d7578f4995b03ac6faba3ec739" member_type: NON_VOTER last_known_addr { host: "127.6.117.197" port: 44467 } attrs { promote: true } }
I20240502 01:59:37.901206 10441 raft_consensus.cc:2932] T 1851ea6467584b0ea32a1f68696803d0 P d6ff17402670432da6eeb5da2af7b4b6 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 29 to 30, NON_VOTER d52d70d7578f4995b03ac6faba3ec739 (127.6.117.197) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } attrs { replace: true } } peers { permanent_uuid: "d6ff17402670432da6eeb5da2af7b4b6" member_type: VOTER last_known_addr { host: "127.6.117.196" port: 34337 } attrs { promote: false } } peers { permanent_uuid: "d52d70d7578f4995b03ac6faba3ec739" member_type: NON_VOTER last_known_addr { host: "127.6.117.197" port: 44467 } attrs { promote: true } } }
I20240502 01:59:37.902532 10053 raft_consensus.cc:1261] T 4885222f238c4d8ea20ebc4c7e171694 P bdbecaeb8f214529b2012a9c003de2ff [term 2 FOLLOWER]: Refusing update from remote peer e77d1c0d41034d1eb5a036dc17c071f5: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240502 01:59:37.902463 9953 catalog_manager.cc:5484] T 1851ea6467584b0ea32a1f68696803d0 P 4b7f7ba2ac084e829c4304c16a94894f reported cstate change: config changed from index 29 to 30, NON_VOTER d52d70d7578f4995b03ac6faba3ec739 (127.6.117.197) added. New cstate: current_term: 2 leader_uuid: "bdbecaeb8f214529b2012a9c003de2ff" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } attrs { replace: true } } peers { permanent_uuid: "d6ff17402670432da6eeb5da2af7b4b6" member_type: VOTER last_known_addr { host: "127.6.117.196" port: 34337 } attrs { promote: false } } peers { permanent_uuid: "d52d70d7578f4995b03ac6faba3ec739" member_type: NON_VOTER last_known_addr { host: "127.6.117.197" port: 44467 } attrs { promote: true } } }
I20240502 01:59:37.902866 10131 raft_consensus.cc:1261] T 4885222f238c4d8ea20ebc4c7e171694 P 4b7f7ba2ac084e829c4304c16a94894f [term 2 FOLLOWER]: Refusing update from remote peer e77d1c0d41034d1eb5a036dc17c071f5: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240502 01:59:37.903231 10789 consensus_queue.cc:1035] T 4885222f238c4d8ea20ebc4c7e171694 P e77d1c0d41034d1eb5a036dc17c071f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 01:59:37.904130 10789 consensus_queue.cc:1035] T 4885222f238c4d8ea20ebc4c7e171694 P e77d1c0d41034d1eb5a036dc17c071f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 01:59:37.905562 10790 raft_consensus.cc:2932] T 4885222f238c4d8ea20ebc4c7e171694 P e77d1c0d41034d1eb5a036dc17c071f5 [term 2 LEADER]: Committing config change with OpId 2.27: config changed from index -1 to 27, NON_VOTER d52d70d7578f4995b03ac6faba3ec739 (127.6.117.197) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } attrs { replace: true } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "d52d70d7578f4995b03ac6faba3ec739" member_type: NON_VOTER last_known_addr { host: "127.6.117.197" port: 44467 } attrs { promote: true } } }
I20240502 01:59:37.905900 10053 raft_consensus.cc:2932] T 4885222f238c4d8ea20ebc4c7e171694 P bdbecaeb8f214529b2012a9c003de2ff [term 2 FOLLOWER]: Committing config change with OpId 2.27: config changed from index -1 to 27, NON_VOTER d52d70d7578f4995b03ac6faba3ec739 (127.6.117.197) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } attrs { replace: true } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "d52d70d7578f4995b03ac6faba3ec739" member_type: NON_VOTER last_known_addr { host: "127.6.117.197" port: 44467 } attrs { promote: true } } }
I20240502 01:59:37.908088 10131 consensus_queue.cc:237] T 19ff9c216a094fde86ea9f9648bf08cb P 4b7f7ba2ac084e829c4304c16a94894f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "d52d70d7578f4995b03ac6faba3ec739" member_type: VOTER last_known_addr { host: "127.6.117.197" port: 44467 } attrs { promote: false } } peers { permanent_uuid: "d6ff17402670432da6eeb5da2af7b4b6" member_type: NON_VOTER last_known_addr { host: "127.6.117.196" port: 34337 } attrs { promote: true } }
W20240502 01:59:37.909395 10171 consensus_peers.cc:485] T 4885222f238c4d8ea20ebc4c7e171694 P e77d1c0d41034d1eb5a036dc17c071f5 -> Peer d52d70d7578f4995b03ac6faba3ec739 (127.6.117.197:44467): Couldn't send request to peer d52d70d7578f4995b03ac6faba3ec739. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4885222f238c4d8ea20ebc4c7e171694. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:37.909433 10517 raft_consensus.cc:1261] T 19ff9c216a094fde86ea9f9648bf08cb P d52d70d7578f4995b03ac6faba3ec739 [term 1 FOLLOWER]: Refusing update from remote peer 4b7f7ba2ac084e829c4304c16a94894f: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
W20240502 01:59:37.909793 10097 consensus_peers.cc:485] T 19ff9c216a094fde86ea9f9648bf08cb P 4b7f7ba2ac084e829c4304c16a94894f -> Peer d6ff17402670432da6eeb5da2af7b4b6 (127.6.117.196:34337): Couldn't send request to peer d6ff17402670432da6eeb5da2af7b4b6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 19ff9c216a094fde86ea9f9648bf08cb. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:37.909812 10796 consensus_queue.cc:1035] T 19ff9c216a094fde86ea9f9648bf08cb P 4b7f7ba2ac084e829c4304c16a94894f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d52d70d7578f4995b03ac6faba3ec739" member_type: VOTER last_known_addr { host: "127.6.117.197" port: 44467 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 01:59:37.910171 10200 raft_consensus.cc:1261] T 19ff9c216a094fde86ea9f9648bf08cb P e77d1c0d41034d1eb5a036dc17c071f5 [term 1 FOLLOWER]: Refusing update from remote peer 4b7f7ba2ac084e829c4304c16a94894f: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240502 01:59:37.910730 9947 catalog_manager.cc:5484] T 4885222f238c4d8ea20ebc4c7e171694 P e77d1c0d41034d1eb5a036dc17c071f5 reported cstate change: config changed from index -1 to 27, NON_VOTER d52d70d7578f4995b03ac6faba3ec739 (127.6.117.197) added. New cstate: current_term: 2 leader_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d52d70d7578f4995b03ac6faba3ec739" member_type: NON_VOTER last_known_addr { host: "127.6.117.197" port: 44467 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:59:37.911325 10797 consensus_queue.cc:1035] T 19ff9c216a094fde86ea9f9648bf08cb P 4b7f7ba2ac084e829c4304c16a94894f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 01:59:37.911712 10731 raft_consensus.cc:2932] T 19ff9c216a094fde86ea9f9648bf08cb P 4b7f7ba2ac084e829c4304c16a94894f [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index 27 to 28, NON_VOTER d6ff17402670432da6eeb5da2af7b4b6 (127.6.117.196) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "d52d70d7578f4995b03ac6faba3ec739" member_type: VOTER last_known_addr { host: "127.6.117.197" port: 44467 } attrs { promote: false } } peers { permanent_uuid: "d6ff17402670432da6eeb5da2af7b4b6" member_type: NON_VOTER last_known_addr { host: "127.6.117.196" port: 34337 } attrs { promote: true } } }
I20240502 01:59:37.911947 10129 raft_consensus.cc:2932] T 4885222f238c4d8ea20ebc4c7e171694 P 4b7f7ba2ac084e829c4304c16a94894f [term 2 FOLLOWER]: Committing config change with OpId 2.27: config changed from index -1 to 27, NON_VOTER d52d70d7578f4995b03ac6faba3ec739 (127.6.117.197) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } attrs { replace: true } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "d52d70d7578f4995b03ac6faba3ec739" member_type: NON_VOTER last_known_addr { host: "127.6.117.197" port: 44467 } attrs { promote: true } } }
I20240502 01:59:37.913781 10517 raft_consensus.cc:2932] T 19ff9c216a094fde86ea9f9648bf08cb P d52d70d7578f4995b03ac6faba3ec739 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, NON_VOTER d6ff17402670432da6eeb5da2af7b4b6 (127.6.117.196) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "d52d70d7578f4995b03ac6faba3ec739" member_type: VOTER last_known_addr { host: "127.6.117.197" port: 44467 } attrs { promote: false } } peers { permanent_uuid: "d6ff17402670432da6eeb5da2af7b4b6" member_type: NON_VOTER last_known_addr { host: "127.6.117.196" port: 34337 } attrs { promote: true } } }
I20240502 01:59:37.914533 10053 consensus_queue.cc:237] T 39f1842ff1494d78bfa0811a799c4bb8 P bdbecaeb8f214529b2012a9c003de2ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } attrs { replace: true } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "ae04d48c97204162b0f8a27d81fcec1a" member_type: NON_VOTER last_known_addr { host: "127.6.117.198" port: 33655 } attrs { promote: true } }
I20240502 01:59:37.914515 10200 raft_consensus.cc:2932] T 19ff9c216a094fde86ea9f9648bf08cb P e77d1c0d41034d1eb5a036dc17c071f5 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, NON_VOTER d6ff17402670432da6eeb5da2af7b4b6 (127.6.117.196) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "d52d70d7578f4995b03ac6faba3ec739" member_type: VOTER last_known_addr { host: "127.6.117.197" port: 44467 } attrs { promote: false } } peers { permanent_uuid: "d6ff17402670432da6eeb5da2af7b4b6" member_type: NON_VOTER last_known_addr { host: "127.6.117.196" port: 34337 } attrs { promote: true } } }
I20240502 01:59:37.915942 10202 raft_consensus.cc:1261] T 39f1842ff1494d78bfa0811a799c4bb8 P e77d1c0d41034d1eb5a036dc17c071f5 [term 1 FOLLOWER]: Refusing update from remote peer bdbecaeb8f214529b2012a9c003de2ff: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240502 01:59:37.916293 10129 raft_consensus.cc:1261] T 39f1842ff1494d78bfa0811a799c4bb8 P 4b7f7ba2ac084e829c4304c16a94894f [term 1 FOLLOWER]: Refusing update from remote peer bdbecaeb8f214529b2012a9c003de2ff: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
W20240502 01:59:37.915963 10021 consensus_peers.cc:485] T 39f1842ff1494d78bfa0811a799c4bb8 P bdbecaeb8f214529b2012a9c003de2ff -> Peer ae04d48c97204162b0f8a27d81fcec1a (127.6.117.198:33655): Couldn't send request to peer ae04d48c97204162b0f8a27d81fcec1a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39f1842ff1494d78bfa0811a799c4bb8. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:37.916628 10739 consensus_queue.cc:1035] T 39f1842ff1494d78bfa0811a799c4bb8 P bdbecaeb8f214529b2012a9c003de2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 01:59:37.916592 9953 catalog_manager.cc:5484] T 19ff9c216a094fde86ea9f9648bf08cb P 4b7f7ba2ac084e829c4304c16a94894f reported cstate change: config changed from index 27 to 28, NON_VOTER d6ff17402670432da6eeb5da2af7b4b6 (127.6.117.196) added. New cstate: current_term: 1 leader_uuid: "4b7f7ba2ac084e829c4304c16a94894f" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d52d70d7578f4995b03ac6faba3ec739" member_type: VOTER last_known_addr { host: "127.6.117.197" port: 44467 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6ff17402670432da6eeb5da2af7b4b6" member_type: NON_VOTER last_known_addr { host: "127.6.117.196" port: 34337 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:59:37.916939 10739 consensus_queue.cc:1035] T 39f1842ff1494d78bfa0811a799c4bb8 P bdbecaeb8f214529b2012a9c003de2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 01:59:37.918658 10739 raft_consensus.cc:2932] T 39f1842ff1494d78bfa0811a799c4bb8 P bdbecaeb8f214529b2012a9c003de2ff [term 1 LEADER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER ae04d48c97204162b0f8a27d81fcec1a (127.6.117.198) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } attrs { replace: true } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "ae04d48c97204162b0f8a27d81fcec1a" member_type: NON_VOTER last_known_addr { host: "127.6.117.198" port: 33655 } attrs { promote: true } } }
I20240502 01:59:37.919051 10202 raft_consensus.cc:2932] T 39f1842ff1494d78bfa0811a799c4bb8 P e77d1c0d41034d1eb5a036dc17c071f5 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER ae04d48c97204162b0f8a27d81fcec1a (127.6.117.198) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } attrs { replace: true } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "ae04d48c97204162b0f8a27d81fcec1a" member_type: NON_VOTER last_known_addr { host: "127.6.117.198" port: 33655 } attrs { promote: true } } }
I20240502 01:59:37.921182 9947 catalog_manager.cc:5484] T 39f1842ff1494d78bfa0811a799c4bb8 P bdbecaeb8f214529b2012a9c003de2ff reported cstate change: config changed from index -1 to 26, NON_VOTER ae04d48c97204162b0f8a27d81fcec1a (127.6.117.198) added. New cstate: current_term: 1 leader_uuid: "bdbecaeb8f214529b2012a9c003de2ff" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae04d48c97204162b0f8a27d81fcec1a" member_type: NON_VOTER last_known_addr { host: "127.6.117.198" port: 33655 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:59:37.921208 10129 raft_consensus.cc:2932] T 39f1842ff1494d78bfa0811a799c4bb8 P 4b7f7ba2ac084e829c4304c16a94894f [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER ae04d48c97204162b0f8a27d81fcec1a (127.6.117.198) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } attrs { replace: true } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "ae04d48c97204162b0f8a27d81fcec1a" member_type: NON_VOTER last_known_addr { host: "127.6.117.198" port: 33655 } attrs { promote: true } } }
I20240502 01:59:37.964546 10008 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:59:37.966446 10202 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "13d726dd71b14e84bc061f537f81e91d"
dest_uuid: "e77d1c0d41034d1eb5a036dc17c071f5"
mode: GRACEFUL
new_leader_uuid: "bdbecaeb8f214529b2012a9c003de2ff"
from {username='slave'} at 127.0.0.1:48104
I20240502 01:59:37.966610 10202 raft_consensus.cc:595] T 13d726dd71b14e84bc061f537f81e91d P e77d1c0d41034d1eb5a036dc17c071f5 [term 2 LEADER]: Received request to transfer leadership to TS bdbecaeb8f214529b2012a9c003de2ff
I20240502 01:59:37.967298 10053 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1851ea6467584b0ea32a1f68696803d0"
dest_uuid: "bdbecaeb8f214529b2012a9c003de2ff"
mode: GRACEFUL
new_leader_uuid: "4b7f7ba2ac084e829c4304c16a94894f"
from {username='slave'} at 127.0.0.1:44420
I20240502 01:59:37.967408 10053 raft_consensus.cc:595] T 1851ea6467584b0ea32a1f68696803d0 P bdbecaeb8f214529b2012a9c003de2ff [term 2 LEADER]: Received request to transfer leadership to TS 4b7f7ba2ac084e829c4304c16a94894f
I20240502 01:59:37.968331 10129 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "73ea0bfb40ad43a68a6289ff1b35d005"
dest_uuid: "4b7f7ba2ac084e829c4304c16a94894f"
mode: GRACEFUL
new_leader_uuid: "d6ff17402670432da6eeb5da2af7b4b6"
from {username='slave'} at 127.0.0.1:52630
I20240502 01:59:37.968443 10129 raft_consensus.cc:595] T 73ea0bfb40ad43a68a6289ff1b35d005 P 4b7f7ba2ac084e829c4304c16a94894f [term 1 LEADER]: Received request to transfer leadership to TS d6ff17402670432da6eeb5da2af7b4b6
I20240502 01:59:37.969097 10054 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "abf7d8b411bb46b68ac9ae77d74f8e2b"
dest_uuid: "bdbecaeb8f214529b2012a9c003de2ff"
mode: GRACEFUL
new_leader_uuid: "4b7f7ba2ac084e829c4304c16a94894f"
from {username='slave'} at 127.0.0.1:44420
I20240502 01:59:37.969200 10054 raft_consensus.cc:595] T abf7d8b411bb46b68ac9ae77d74f8e2b P bdbecaeb8f214529b2012a9c003de2ff [term 1 LEADER]: Received request to transfer leadership to TS 4b7f7ba2ac084e829c4304c16a94894f
I20240502 01:59:37.969779 10129 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b4409766ce994f698c31aede604a6927"
dest_uuid: "4b7f7ba2ac084e829c4304c16a94894f"
mode: GRACEFUL
new_leader_uuid: "e77d1c0d41034d1eb5a036dc17c071f5"
from {username='slave'} at 127.0.0.1:52630
I20240502 01:59:37.969877 10129 raft_consensus.cc:595] T b4409766ce994f698c31aede604a6927 P 4b7f7ba2ac084e829c4304c16a94894f [term 3 LEADER]: Received request to transfer leadership to TS e77d1c0d41034d1eb5a036dc17c071f5
I20240502 01:59:37.970139 10008 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 5
I20240502 01:59:37.970290 10008 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:59:38.098567 10731 raft_consensus.cc:979] T b4409766ce994f698c31aede604a6927 P 4b7f7ba2ac084e829c4304c16a94894f: : Instructing follower e77d1c0d41034d1eb5a036dc17c071f5 to start an election
I20240502 01:59:38.098714 10797 raft_consensus.cc:1067] T b4409766ce994f698c31aede604a6927 P 4b7f7ba2ac084e829c4304c16a94894f [term 3 LEADER]: Signalling peer e77d1c0d41034d1eb5a036dc17c071f5 to start an election
I20240502 01:59:38.099141 10202 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b4409766ce994f698c31aede604a6927"
dest_uuid: "e77d1c0d41034d1eb5a036dc17c071f5"
from {username='slave'} at 127.0.0.1:48076
I20240502 01:59:38.099273 10202 raft_consensus.cc:482] T b4409766ce994f698c31aede604a6927 P e77d1c0d41034d1eb5a036dc17c071f5 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:59:38.099339 10202 raft_consensus.cc:3037] T b4409766ce994f698c31aede604a6927 P e77d1c0d41034d1eb5a036dc17c071f5 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:59:38.100351 10202 raft_consensus.cc:504] T b4409766ce994f698c31aede604a6927 P e77d1c0d41034d1eb5a036dc17c071f5 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "ae04d48c97204162b0f8a27d81fcec1a" member_type: VOTER last_known_addr { host: "127.6.117.198" port: 33655 } attrs { promote: false } }
I20240502 01:59:38.100703 10202 leader_election.cc:288] T b4409766ce994f698c31aede604a6927 P e77d1c0d41034d1eb5a036dc17c071f5 [CANDIDATE]: Term 4 election: Requested vote from peers 4b7f7ba2ac084e829c4304c16a94894f (127.6.117.194:46805), ae04d48c97204162b0f8a27d81fcec1a (127.6.117.198:33655)
I20240502 01:59:38.100934 10129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4409766ce994f698c31aede604a6927" candidate_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 29 } } ignore_live_leader: true dest_uuid: "4b7f7ba2ac084e829c4304c16a94894f"
I20240502 01:59:38.101045 10129 raft_consensus.cc:3032] T b4409766ce994f698c31aede604a6927 P 4b7f7ba2ac084e829c4304c16a94894f [term 3 LEADER]: Stepping down as leader of term 3
I20240502 01:59:38.101091 10129 raft_consensus.cc:729] T b4409766ce994f698c31aede604a6927 P 4b7f7ba2ac084e829c4304c16a94894f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4b7f7ba2ac084e829c4304c16a94894f, State: Running, Role: LEADER
I20240502 01:59:38.101106 10590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4409766ce994f698c31aede604a6927" candidate_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 29 } } ignore_live_leader: true dest_uuid: "ae04d48c97204162b0f8a27d81fcec1a"
I20240502 01:59:38.101222 10590 raft_consensus.cc:3037] T b4409766ce994f698c31aede604a6927 P ae04d48c97204162b0f8a27d81fcec1a [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:59:38.101224 10129 consensus_queue.cc:260] T b4409766ce994f698c31aede604a6927 P 4b7f7ba2ac084e829c4304c16a94894f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 3.29, Last appended by leader: 29, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "ae04d48c97204162b0f8a27d81fcec1a" member_type: VOTER last_known_addr { host: "127.6.117.198" port: 33655 } attrs { promote: false } }
I20240502 01:59:38.101408 10129 raft_consensus.cc:3037] T b4409766ce994f698c31aede604a6927 P 4b7f7ba2ac084e829c4304c16a94894f [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:59:38.102016 10590 raft_consensus.cc:2454] T b4409766ce994f698c31aede604a6927 P ae04d48c97204162b0f8a27d81fcec1a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e77d1c0d41034d1eb5a036dc17c071f5 in term 4.
I20240502 01:59:38.102056 10129 raft_consensus.cc:2454] T b4409766ce994f698c31aede604a6927 P 4b7f7ba2ac084e829c4304c16a94894f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e77d1c0d41034d1eb5a036dc17c071f5 in term 4.
I20240502 01:59:38.102319 10173 leader_election.cc:302] T b4409766ce994f698c31aede604a6927 P e77d1c0d41034d1eb5a036dc17c071f5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b7f7ba2ac084e829c4304c16a94894f, e77d1c0d41034d1eb5a036dc17c071f5; no voters:
I20240502 01:59:38.102489 10790 raft_consensus.cc:2781] T b4409766ce994f698c31aede604a6927 P e77d1c0d41034d1eb5a036dc17c071f5 [term 4 FOLLOWER]: Leader election won for term 4
I20240502 01:59:38.102573 10790 raft_consensus.cc:686] T b4409766ce994f698c31aede604a6927 P e77d1c0d41034d1eb5a036dc17c071f5 [term 4 LEADER]: Becoming Leader. State: Replica: e77d1c0d41034d1eb5a036dc17c071f5, State: Running, Role: LEADER
I20240502 01:59:38.102699 10790 consensus_queue.cc:237] T b4409766ce994f698c31aede604a6927 P e77d1c0d41034d1eb5a036dc17c071f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 3.29, Last appended by leader: 29, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } } peers { permanent_uuid: "ae04d48c97204162b0f8a27d81fcec1a" member_type: VOTER last_known_addr { host: "127.6.117.198" port: 33655 } attrs { promote: false } }
I20240502 01:59:38.104229 9948 catalog_manager.cc:5484] T b4409766ce994f698c31aede604a6927 P e77d1c0d41034d1eb5a036dc17c071f5 reported cstate change: term changed from 3 to 4, leader changed from 4b7f7ba2ac084e829c4304c16a94894f (127.6.117.194) to e77d1c0d41034d1eb5a036dc17c071f5 (127.6.117.195). New cstate: current_term: 4 leader_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae04d48c97204162b0f8a27d81fcec1a" member_type: VOTER last_known_addr { host: "127.6.117.198" port: 33655 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:59:38.238164 10725 raft_consensus.cc:979] T 13d726dd71b14e84bc061f537f81e91d P e77d1c0d41034d1eb5a036dc17c071f5: : Instructing follower bdbecaeb8f214529b2012a9c003de2ff to start an election
I20240502 01:59:38.238296 10725 raft_consensus.cc:1067] T 13d726dd71b14e84bc061f537f81e91d P e77d1c0d41034d1eb5a036dc17c071f5 [term 2 LEADER]: Signalling peer bdbecaeb8f214529b2012a9c003de2ff to start an election
I20240502 01:59:38.238713 10053 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "13d726dd71b14e84bc061f537f81e91d"
dest_uuid: "bdbecaeb8f214529b2012a9c003de2ff"
from {username='slave'} at 127.0.0.1:44398
I20240502 01:59:38.238845 10053 raft_consensus.cc:482] T 13d726dd71b14e84bc061f537f81e91d P bdbecaeb8f214529b2012a9c003de2ff [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:59:38.238907 10053 raft_consensus.cc:3037] T 13d726dd71b14e84bc061f537f81e91d P bdbecaeb8f214529b2012a9c003de2ff [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:59:38.239779 10053 raft_consensus.cc:504] T 13d726dd71b14e84bc061f537f81e91d P bdbecaeb8f214529b2012a9c003de2ff [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:38.240099 10053 leader_election.cc:288] T 13d726dd71b14e84bc061f537f81e91d P bdbecaeb8f214529b2012a9c003de2ff [CANDIDATE]: Term 3 election: Requested vote from peers 4b7f7ba2ac084e829c4304c16a94894f (127.6.117.194:46805), e77d1c0d41034d1eb5a036dc17c071f5 (127.6.117.195:43977)
I20240502 01:59:38.240332 10129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13d726dd71b14e84bc061f537f81e91d" candidate_uuid: "bdbecaeb8f214529b2012a9c003de2ff" candidate_term: 3 candidate_status { last_received { term: 2 index: 25 } } ignore_live_leader: true dest_uuid: "4b7f7ba2ac084e829c4304c16a94894f"
I20240502 01:59:38.240435 10129 raft_consensus.cc:3037] T 13d726dd71b14e84bc061f537f81e91d P 4b7f7ba2ac084e829c4304c16a94894f [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:59:38.241250 10129 raft_consensus.cc:2454] T 13d726dd71b14e84bc061f537f81e91d P 4b7f7ba2ac084e829c4304c16a94894f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdbecaeb8f214529b2012a9c003de2ff in term 3.
I20240502 01:59:38.241454 10202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13d726dd71b14e84bc061f537f81e91d" candidate_uuid: "bdbecaeb8f214529b2012a9c003de2ff" candidate_term: 3 candidate_status { last_received { term: 2 index: 25 } } ignore_live_leader: true dest_uuid: "e77d1c0d41034d1eb5a036dc17c071f5"
I20240502 01:59:38.241497 10022 leader_election.cc:302] T 13d726dd71b14e84bc061f537f81e91d P bdbecaeb8f214529b2012a9c003de2ff [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b7f7ba2ac084e829c4304c16a94894f, bdbecaeb8f214529b2012a9c003de2ff; no voters:
I20240502 01:59:38.241545 10202 raft_consensus.cc:3032] T 13d726dd71b14e84bc061f537f81e91d P e77d1c0d41034d1eb5a036dc17c071f5 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:59:38.241686 10202 raft_consensus.cc:729] T 13d726dd71b14e84bc061f537f81e91d P e77d1c0d41034d1eb5a036dc17c071f5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e77d1c0d41034d1eb5a036dc17c071f5, State: Running, Role: LEADER
I20240502 01:59:38.241752 10785 raft_consensus.cc:2781] T 13d726dd71b14e84bc061f537f81e91d P bdbecaeb8f214529b2012a9c003de2ff [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:59:38.241837 10202 consensus_queue.cc:260] T 13d726dd71b14e84bc061f537f81e91d P e77d1c0d41034d1eb5a036dc17c071f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 2.25, Last appended by leader: 25, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:38.242022 10202 raft_consensus.cc:3037] T 13d726dd71b14e84bc061f537f81e91d P e77d1c0d41034d1eb5a036dc17c071f5 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:59:38.242727 10202 raft_consensus.cc:2454] T 13d726dd71b14e84bc061f537f81e91d P e77d1c0d41034d1eb5a036dc17c071f5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdbecaeb8f214529b2012a9c003de2ff in term 3.
I20240502 01:59:38.241844 10785 raft_consensus.cc:686] T 13d726dd71b14e84bc061f537f81e91d P bdbecaeb8f214529b2012a9c003de2ff [term 3 LEADER]: Becoming Leader. State: Replica: bdbecaeb8f214529b2012a9c003de2ff, State: Running, Role: LEADER
I20240502 01:59:38.243006 10785 consensus_queue.cc:237] T 13d726dd71b14e84bc061f537f81e91d P bdbecaeb8f214529b2012a9c003de2ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 2.25, Last appended by leader: 25, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } }
I20240502 01:59:38.244752 9948 catalog_manager.cc:5484] T 13d726dd71b14e84bc061f537f81e91d P bdbecaeb8f214529b2012a9c003de2ff reported cstate change: term changed from 2 to 3, leader changed from e77d1c0d41034d1eb5a036dc17c071f5 (127.6.117.195) to bdbecaeb8f214529b2012a9c003de2ff (127.6.117.193). New cstate: current_term: 3 leader_uuid: "bdbecaeb8f214529b2012a9c003de2ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } health_report { overall_health: UNKNOWN } } }
I20240502 01:59:38.292853 10739 raft_consensus.cc:979] T abf7d8b411bb46b68ac9ae77d74f8e2b P bdbecaeb8f214529b2012a9c003de2ff: : Instructing follower 4b7f7ba2ac084e829c4304c16a94894f to start an election
I20240502 01:59:38.292996 10739 raft_consensus.cc:1067] T abf7d8b411bb46b68ac9ae77d74f8e2b P bdbecaeb8f214529b2012a9c003de2ff [term 1 LEADER]: Signalling peer 4b7f7ba2ac084e829c4304c16a94894f to start an election
I20240502 01:59:38.293365 10129 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "abf7d8b411bb46b68ac9ae77d74f8e2b"
dest_uuid: "4b7f7ba2ac084e829c4304c16a94894f"
from {username='slave'} at 127.0.0.1:52602
I20240502 01:59:38.293503 10129 raft_consensus.cc:482] T abf7d8b411bb46b68ac9ae77d74f8e2b P 4b7f7ba2ac084e829c4304c16a94894f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:59:38.293563 10129 raft_consensus.cc:3037] T abf7d8b411bb46b68ac9ae77d74f8e2b P 4b7f7ba2ac084e829c4304c16a94894f [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:59:38.294549 10129 raft_consensus.cc:504] T abf7d8b411bb46b68ac9ae77d74f8e2b P 4b7f7ba2ac084e829c4304c16a94894f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "ae04d48c97204162b0f8a27d81fcec1a" member_type: NON_VOTER last_known_addr { host: "127.6.117.198" port: 33655 } attrs { promote: true } }
I20240502 01:59:38.294936 10129 leader_election.cc:288] T abf7d8b411bb46b68ac9ae77d74f8e2b P 4b7f7ba2ac084e829c4304c16a94894f [CANDIDATE]: Term 2 election: Requested vote from peers bdbecaeb8f214529b2012a9c003de2ff (127.6.117.193:34597), e77d1c0d41034d1eb5a036dc17c071f5 (127.6.117.195:43977)
I20240502 01:59:38.295184 10054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abf7d8b411bb46b68ac9ae77d74f8e2b" candidate_uuid: "4b7f7ba2ac084e829c4304c16a94894f" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: true dest_uuid: "bdbecaeb8f214529b2012a9c003de2ff"
I20240502 01:59:38.295292 10054 raft_consensus.cc:3032] T abf7d8b411bb46b68ac9ae77d74f8e2b P bdbecaeb8f214529b2012a9c003de2ff [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:59:38.295359 10054 raft_consensus.cc:729] T abf7d8b411bb46b68ac9ae77d74f8e2b P bdbecaeb8f214529b2012a9c003de2ff [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bdbecaeb8f214529b2012a9c003de2ff, State: Running, Role: LEADER
I20240502 01:59:38.295293 10202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abf7d8b411bb46b68ac9ae77d74f8e2b" candidate_uuid: "4b7f7ba2ac084e829c4304c16a94894f" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: true dest_uuid: "e77d1c0d41034d1eb5a036dc17c071f5"
I20240502 01:59:38.295514 10202 raft_consensus.cc:3037] T abf7d8b411bb46b68ac9ae77d74f8e2b P e77d1c0d41034d1eb5a036dc17c071f5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:59:38.295531 10054 consensus_queue.cc:260] T abf7d8b411bb46b68ac9ae77d74f8e2b P bdbecaeb8f214529b2012a9c003de2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "ae04d48c97204162b0f8a27d81fcec1a" member_type: NON_VOTER last_known_addr { host: "127.6.117.198" port: 33655 } attrs { promote: true } }
I20240502 01:59:38.295729 10054 raft_consensus.cc:3037] T abf7d8b411bb46b68ac9ae77d74f8e2b P bdbecaeb8f214529b2012a9c003de2ff [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:59:38.296592 10202 raft_consensus.cc:2454] T abf7d8b411bb46b68ac9ae77d74f8e2b P e77d1c0d41034d1eb5a036dc17c071f5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b7f7ba2ac084e829c4304c16a94894f in term 2.
I20240502 01:59:38.296684 10054 raft_consensus.cc:2454] T abf7d8b411bb46b68ac9ae77d74f8e2b P bdbecaeb8f214529b2012a9c003de2ff [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b7f7ba2ac084e829c4304c16a94894f in term 2.
I20240502 01:59:38.296865 10097 leader_election.cc:302] T abf7d8b411bb46b68ac9ae77d74f8e2b P 4b7f7ba2ac084e829c4304c16a94894f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b7f7ba2ac084e829c4304c16a94894f, e77d1c0d41034d1eb5a036dc17c071f5; no voters:
I20240502 01:59:38.297040 10797 raft_consensus.cc:2781] T abf7d8b411bb46b68ac9ae77d74f8e2b P 4b7f7ba2ac084e829c4304c16a94894f [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:59:38.297138 10797 raft_consensus.cc:686] T abf7d8b411bb46b68ac9ae77d74f8e2b P 4b7f7ba2ac084e829c4304c16a94894f [term 2 LEADER]: Becoming Leader. State: Replica: 4b7f7ba2ac084e829c4304c16a94894f, State: Running, Role: LEADER
I20240502 01:59:38.297333 10797 consensus_queue.cc:237] T abf7d8b411bb46b68ac9ae77d74f8e2b P 4b7f7ba2ac084e829c4304c16a94894f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "ae04d48c97204162b0f8a27d81fcec1a" member_type: NON_VOTER last_known_addr { host: "127.6.117.198" port: 33655 } attrs { promote: true } }
I20240502 01:59:38.298946 9948 catalog_manager.cc:5484] T abf7d8b411bb46b68ac9ae77d74f8e2b P 4b7f7ba2ac084e829c4304c16a94894f reported cstate change: term changed from 1 to 2, leader changed from bdbecaeb8f214529b2012a9c003de2ff (127.6.117.193) to 4b7f7ba2ac084e829c4304c16a94894f (127.6.117.194). New cstate: current_term: 2 leader_uuid: "4b7f7ba2ac084e829c4304c16a94894f" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae04d48c97204162b0f8a27d81fcec1a" member_type: NON_VOTER last_known_addr { host: "127.6.117.198" port: 33655 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:59:38.334520 10797 raft_consensus.cc:979] T 73ea0bfb40ad43a68a6289ff1b35d005 P 4b7f7ba2ac084e829c4304c16a94894f: : Instructing follower d6ff17402670432da6eeb5da2af7b4b6 to start an election
I20240502 01:59:38.334731 10801 raft_consensus.cc:1067] T 73ea0bfb40ad43a68a6289ff1b35d005 P 4b7f7ba2ac084e829c4304c16a94894f [term 1 LEADER]: Signalling peer d6ff17402670432da6eeb5da2af7b4b6 to start an election
I20240502 01:59:38.335224 10441 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "73ea0bfb40ad43a68a6289ff1b35d005"
dest_uuid: "d6ff17402670432da6eeb5da2af7b4b6"
from {username='slave'} at 127.0.0.1:35980
I20240502 01:59:38.335357 10441 raft_consensus.cc:482] T 73ea0bfb40ad43a68a6289ff1b35d005 P d6ff17402670432da6eeb5da2af7b4b6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:59:38.335422 10441 raft_consensus.cc:3037] T 73ea0bfb40ad43a68a6289ff1b35d005 P d6ff17402670432da6eeb5da2af7b4b6 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:59:38.336494 10441 raft_consensus.cc:504] T 73ea0bfb40ad43a68a6289ff1b35d005 P d6ff17402670432da6eeb5da2af7b4b6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "d6ff17402670432da6eeb5da2af7b4b6" member_type: VOTER last_known_addr { host: "127.6.117.196" port: 34337 } attrs { promote: false } }
I20240502 01:59:38.336939 10441 leader_election.cc:288] T 73ea0bfb40ad43a68a6289ff1b35d005 P d6ff17402670432da6eeb5da2af7b4b6 [CANDIDATE]: Term 2 election: Requested vote from peers bdbecaeb8f214529b2012a9c003de2ff (127.6.117.193:34597), 4b7f7ba2ac084e829c4304c16a94894f (127.6.117.194:46805)
I20240502 01:59:38.337595 10129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73ea0bfb40ad43a68a6289ff1b35d005" candidate_uuid: "d6ff17402670432da6eeb5da2af7b4b6" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "4b7f7ba2ac084e829c4304c16a94894f"
I20240502 01:59:38.337785 10129 raft_consensus.cc:3032] T 73ea0bfb40ad43a68a6289ff1b35d005 P 4b7f7ba2ac084e829c4304c16a94894f [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:59:38.337832 10129 raft_consensus.cc:729] T 73ea0bfb40ad43a68a6289ff1b35d005 P 4b7f7ba2ac084e829c4304c16a94894f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4b7f7ba2ac084e829c4304c16a94894f, State: Running, Role: LEADER
I20240502 01:59:38.337989 10129 consensus_queue.cc:260] T 73ea0bfb40ad43a68a6289ff1b35d005 P 4b7f7ba2ac084e829c4304c16a94894f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "d6ff17402670432da6eeb5da2af7b4b6" member_type: VOTER last_known_addr { host: "127.6.117.196" port: 34337 } attrs { promote: false } }
I20240502 01:59:38.338168 10129 raft_consensus.cc:3037] T 73ea0bfb40ad43a68a6289ff1b35d005 P 4b7f7ba2ac084e829c4304c16a94894f [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:59:38.339291 10129 raft_consensus.cc:2454] T 73ea0bfb40ad43a68a6289ff1b35d005 P 4b7f7ba2ac084e829c4304c16a94894f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6ff17402670432da6eeb5da2af7b4b6 in term 2.
I20240502 01:59:38.339614 10406 leader_election.cc:302] T 73ea0bfb40ad43a68a6289ff1b35d005 P d6ff17402670432da6eeb5da2af7b4b6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b7f7ba2ac084e829c4304c16a94894f, d6ff17402670432da6eeb5da2af7b4b6; no voters:
I20240502 01:59:38.339805 10792 raft_consensus.cc:2781] T 73ea0bfb40ad43a68a6289ff1b35d005 P d6ff17402670432da6eeb5da2af7b4b6 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:59:38.340059 10792 raft_consensus.cc:686] T 73ea0bfb40ad43a68a6289ff1b35d005 P d6ff17402670432da6eeb5da2af7b4b6 [term 2 LEADER]: Becoming Leader. State: Replica: d6ff17402670432da6eeb5da2af7b4b6, State: Running, Role: LEADER
I20240502 01:59:38.340198 10792 consensus_queue.cc:237] T 73ea0bfb40ad43a68a6289ff1b35d005 P d6ff17402670432da6eeb5da2af7b4b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } } peers { permanent_uuid: "d6ff17402670432da6eeb5da2af7b4b6" member_type: VOTER last_known_addr { host: "127.6.117.196" port: 34337 } attrs { promote: false } }
I20240502 01:59:38.341302 10054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73ea0bfb40ad43a68a6289ff1b35d005" candidate_uuid: "d6ff17402670432da6eeb5da2af7b4b6" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "bdbecaeb8f214529b2012a9c003de2ff"
I20240502 01:59:38.341410 10054 raft_consensus.cc:3037] T 73ea0bfb40ad43a68a6289ff1b35d005 P bdbecaeb8f214529b2012a9c003de2ff [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:59:38.341912 9948 catalog_manager.cc:5484] T 73ea0bfb40ad43a68a6289ff1b35d005 P d6ff17402670432da6eeb5da2af7b4b6 reported cstate change: term changed from 1 to 2, leader changed from 4b7f7ba2ac084e829c4304c16a94894f (127.6.117.194) to d6ff17402670432da6eeb5da2af7b4b6 (127.6.117.196). New cstate: current_term: 2 leader_uuid: "d6ff17402670432da6eeb5da2af7b4b6" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6ff17402670432da6eeb5da2af7b4b6" member_type: VOTER last_known_addr { host: "127.6.117.196" port: 34337 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:59:38.342430 10054 raft_consensus.cc:2454] T 73ea0bfb40ad43a68a6289ff1b35d005 P bdbecaeb8f214529b2012a9c003de2ff [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6ff17402670432da6eeb5da2af7b4b6 in term 2.
I20240502 01:59:38.389880 10785 raft_consensus.cc:979] T 1851ea6467584b0ea32a1f68696803d0 P bdbecaeb8f214529b2012a9c003de2ff: : Instructing follower 4b7f7ba2ac084e829c4304c16a94894f to start an election
I20240502 01:59:38.390012 10785 raft_consensus.cc:1067] T 1851ea6467584b0ea32a1f68696803d0 P bdbecaeb8f214529b2012a9c003de2ff [term 2 LEADER]: Signalling peer 4b7f7ba2ac084e829c4304c16a94894f to start an election
I20240502 01:59:38.390393 10129 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1851ea6467584b0ea32a1f68696803d0"
dest_uuid: "4b7f7ba2ac084e829c4304c16a94894f"
from {username='slave'} at 127.0.0.1:52602
I20240502 01:59:38.390563 10129 raft_consensus.cc:482] T 1851ea6467584b0ea32a1f68696803d0 P 4b7f7ba2ac084e829c4304c16a94894f [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:59:38.390637 10129 raft_consensus.cc:3037] T 1851ea6467584b0ea32a1f68696803d0 P 4b7f7ba2ac084e829c4304c16a94894f [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:59:38.391875 10129 raft_consensus.cc:504] T 1851ea6467584b0ea32a1f68696803d0 P 4b7f7ba2ac084e829c4304c16a94894f [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } attrs { replace: true } } peers { permanent_uuid: "d6ff17402670432da6eeb5da2af7b4b6" member_type: VOTER last_known_addr { host: "127.6.117.196" port: 34337 } attrs { promote: false } } peers { permanent_uuid: "d52d70d7578f4995b03ac6faba3ec739" member_type: NON_VOTER last_known_addr { host: "127.6.117.197" port: 44467 } attrs { promote: true } }
I20240502 01:59:38.392302 10129 leader_election.cc:288] T 1851ea6467584b0ea32a1f68696803d0 P 4b7f7ba2ac084e829c4304c16a94894f [CANDIDATE]: Term 3 election: Requested vote from peers bdbecaeb8f214529b2012a9c003de2ff (127.6.117.193:34597), d6ff17402670432da6eeb5da2af7b4b6 (127.6.117.196:34337)
I20240502 01:59:38.392529 10054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1851ea6467584b0ea32a1f68696803d0" candidate_uuid: "4b7f7ba2ac084e829c4304c16a94894f" candidate_term: 3 candidate_status { last_received { term: 2 index: 30 } } ignore_live_leader: true dest_uuid: "bdbecaeb8f214529b2012a9c003de2ff"
I20240502 01:59:38.392643 10054 raft_consensus.cc:3032] T 1851ea6467584b0ea32a1f68696803d0 P bdbecaeb8f214529b2012a9c003de2ff [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:59:38.392714 10054 raft_consensus.cc:729] T 1851ea6467584b0ea32a1f68696803d0 P bdbecaeb8f214529b2012a9c003de2ff [term 2 LEADER]: Becoming Follower/Learner. State: Replica: bdbecaeb8f214529b2012a9c003de2ff, State: Running, Role: LEADER
I20240502 01:59:38.392729 10441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1851ea6467584b0ea32a1f68696803d0" candidate_uuid: "4b7f7ba2ac084e829c4304c16a94894f" candidate_term: 3 candidate_status { last_received { term: 2 index: 30 } } ignore_live_leader: true dest_uuid: "d6ff17402670432da6eeb5da2af7b4b6"
I20240502 01:59:38.392839 10441 raft_consensus.cc:3037] T 1851ea6467584b0ea32a1f68696803d0 P d6ff17402670432da6eeb5da2af7b4b6 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:59:38.392880 10054 consensus_queue.cc:260] T 1851ea6467584b0ea32a1f68696803d0 P bdbecaeb8f214529b2012a9c003de2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } attrs { replace: true } } peers { permanent_uuid: "d6ff17402670432da6eeb5da2af7b4b6" member_type: VOTER last_known_addr { host: "127.6.117.196" port: 34337 } attrs { promote: false } } peers { permanent_uuid: "d52d70d7578f4995b03ac6faba3ec739" member_type: NON_VOTER last_known_addr { host: "127.6.117.197" port: 44467 } attrs { promote: true } }
I20240502 01:59:38.393070 10054 raft_consensus.cc:3037] T 1851ea6467584b0ea32a1f68696803d0 P bdbecaeb8f214529b2012a9c003de2ff [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:59:38.393757 10441 raft_consensus.cc:2454] T 1851ea6467584b0ea32a1f68696803d0 P d6ff17402670432da6eeb5da2af7b4b6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b7f7ba2ac084e829c4304c16a94894f in term 3.
I20240502 01:59:38.393811 10054 raft_consensus.cc:2454] T 1851ea6467584b0ea32a1f68696803d0 P bdbecaeb8f214529b2012a9c003de2ff [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b7f7ba2ac084e829c4304c16a94894f in term 3.
I20240502 01:59:38.394045 10097 leader_election.cc:302] T 1851ea6467584b0ea32a1f68696803d0 P 4b7f7ba2ac084e829c4304c16a94894f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b7f7ba2ac084e829c4304c16a94894f, d6ff17402670432da6eeb5da2af7b4b6; no voters:
I20240502 01:59:38.394214 10801 raft_consensus.cc:2781] T 1851ea6467584b0ea32a1f68696803d0 P 4b7f7ba2ac084e829c4304c16a94894f [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:59:38.394296 10801 raft_consensus.cc:686] T 1851ea6467584b0ea32a1f68696803d0 P 4b7f7ba2ac084e829c4304c16a94894f [term 3 LEADER]: Becoming Leader. State: Replica: 4b7f7ba2ac084e829c4304c16a94894f, State: Running, Role: LEADER
I20240502 01:59:38.394438 10801 consensus_queue.cc:237] T 1851ea6467584b0ea32a1f68696803d0 P 4b7f7ba2ac084e829c4304c16a94894f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } attrs { replace: true } } peers { permanent_uuid: "d6ff17402670432da6eeb5da2af7b4b6" member_type: VOTER last_known_addr { host: "127.6.117.196" port: 34337 } attrs { promote: false } } peers { permanent_uuid: "d52d70d7578f4995b03ac6faba3ec739" member_type: NON_VOTER last_known_addr { host: "127.6.117.197" port: 44467 } attrs { promote: true } }
I20240502 01:59:38.396083 9948 catalog_manager.cc:5484] T 1851ea6467584b0ea32a1f68696803d0 P 4b7f7ba2ac084e829c4304c16a94894f reported cstate change: term changed from 2 to 3, leader changed from bdbecaeb8f214529b2012a9c003de2ff (127.6.117.193) to 4b7f7ba2ac084e829c4304c16a94894f (127.6.117.194). New cstate: current_term: 3 leader_uuid: "4b7f7ba2ac084e829c4304c16a94894f" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6ff17402670432da6eeb5da2af7b4b6" member_type: VOTER last_known_addr { host: "127.6.117.196" port: 34337 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d52d70d7578f4995b03ac6faba3ec739" member_type: NON_VOTER last_known_addr { host: "127.6.117.197" port: 44467 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:59:38.401870 10129 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1851ea6467584b0ea32a1f68696803d0"
dest_uuid: "4b7f7ba2ac084e829c4304c16a94894f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52598
I20240502 01:59:38.402010 10129 raft_consensus.cc:595] T 1851ea6467584b0ea32a1f68696803d0 P 4b7f7ba2ac084e829c4304c16a94894f [term 3 LEADER]: Received request to transfer leadership
I20240502 01:59:38.407090 10129 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1851ea6467584b0ea32a1f68696803d0"
dest_uuid: "4b7f7ba2ac084e829c4304c16a94894f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52598
I20240502 01:59:38.407249 10129 raft_consensus.cc:595] T 1851ea6467584b0ea32a1f68696803d0 P 4b7f7ba2ac084e829c4304c16a94894f [term 3 LEADER]: Received request to transfer leadership
I20240502 01:59:38.408053 10129 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1851ea6467584b0ea32a1f68696803d0"
dest_uuid: "4b7f7ba2ac084e829c4304c16a94894f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52598
I20240502 01:59:38.408155 10129 raft_consensus.cc:595] T 1851ea6467584b0ea32a1f68696803d0 P 4b7f7ba2ac084e829c4304c16a94894f [term 3 LEADER]: Received request to transfer leadership
W20240502 01:59:38.408488 10003 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for 1851ea6467584b0ea32a1f68696803d0 already in progress
W20240502 01:59:38.408545 10003 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for 1851ea6467584b0ea32a1f68696803d0 already in progress
I20240502 01:59:38.414103 10815 ts_tablet_manager.cc:902] T e7a8fc5afcab4b1db4ebf245b66df4be P d6ff17402670432da6eeb5da2af7b4b6: Initiating tablet copy from peer ae04d48c97204162b0f8a27d81fcec1a (127.6.117.198:33655)
I20240502 01:59:38.414582 10815 tablet_copy_client.cc:308] T e7a8fc5afcab4b1db4ebf245b66df4be P d6ff17402670432da6eeb5da2af7b4b6: tablet copy: Beginning tablet copy session from remote peer at address 127.6.117.198:33655
I20240502 01:59:38.418190 10602 tablet_copy_service.cc:140] P ae04d48c97204162b0f8a27d81fcec1a: Received BeginTabletCopySession request for tablet e7a8fc5afcab4b1db4ebf245b66df4be from peer d6ff17402670432da6eeb5da2af7b4b6 ({username='slave'} at 127.0.0.1:36556)
I20240502 01:59:38.418308 10602 tablet_copy_service.cc:161] P ae04d48c97204162b0f8a27d81fcec1a: Beginning new tablet copy session on tablet e7a8fc5afcab4b1db4ebf245b66df4be from peer d6ff17402670432da6eeb5da2af7b4b6 at {username='slave'} at 127.0.0.1:36556: session id = d6ff17402670432da6eeb5da2af7b4b6-e7a8fc5afcab4b1db4ebf245b66df4be
I20240502 01:59:38.419129 10602 tablet_copy_source_session.cc:205] T e7a8fc5afcab4b1db4ebf245b66df4be P ae04d48c97204162b0f8a27d81fcec1a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:59:38.419842 10815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7a8fc5afcab4b1db4ebf245b66df4be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:38.422027 10815 tablet_copy_client.cc:783] T e7a8fc5afcab4b1db4ebf245b66df4be P d6ff17402670432da6eeb5da2af7b4b6: tablet copy: Starting download of 0 data blocks...
I20240502 01:59:38.422212 10815 tablet_copy_client.cc:647] T e7a8fc5afcab4b1db4ebf245b66df4be P d6ff17402670432da6eeb5da2af7b4b6: tablet copy: Starting download of 1 WAL segments...
I20240502 01:59:38.423175 10819 tablet_copy_client.cc:1002] T e7a8fc5afcab4b1db4ebf245b66df4be P d6ff17402670432da6eeb5da2af7b4b6: tablet copy: Injecting latency into file download: 2000
I20240502 01:59:38.450951 10821 ts_tablet_manager.cc:902] T 19ff9c216a094fde86ea9f9648bf08cb P d6ff17402670432da6eeb5da2af7b4b6: Initiating tablet copy from peer 4b7f7ba2ac084e829c4304c16a94894f (127.6.117.194:46805)
I20240502 01:59:38.451548 10821 tablet_copy_client.cc:308] T 19ff9c216a094fde86ea9f9648bf08cb P d6ff17402670432da6eeb5da2af7b4b6: tablet copy: Beginning tablet copy session from remote peer at address 127.6.117.194:46805
I20240502 01:59:38.451921 10143 tablet_copy_service.cc:140] P 4b7f7ba2ac084e829c4304c16a94894f: Received BeginTabletCopySession request for tablet 19ff9c216a094fde86ea9f9648bf08cb from peer d6ff17402670432da6eeb5da2af7b4b6 ({username='slave'} at 127.0.0.1:52658)
I20240502 01:59:38.452021 10143 tablet_copy_service.cc:161] P 4b7f7ba2ac084e829c4304c16a94894f: Beginning new tablet copy session on tablet 19ff9c216a094fde86ea9f9648bf08cb from peer d6ff17402670432da6eeb5da2af7b4b6 at {username='slave'} at 127.0.0.1:52658: session id = d6ff17402670432da6eeb5da2af7b4b6-19ff9c216a094fde86ea9f9648bf08cb
I20240502 01:59:38.453068 10143 tablet_copy_source_session.cc:205] T 19ff9c216a094fde86ea9f9648bf08cb P 4b7f7ba2ac084e829c4304c16a94894f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:59:38.453837 10821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19ff9c216a094fde86ea9f9648bf08cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:38.455986 10821 tablet_copy_client.cc:783] T 19ff9c216a094fde86ea9f9648bf08cb P d6ff17402670432da6eeb5da2af7b4b6: tablet copy: Starting download of 0 data blocks...
I20240502 01:59:38.456126 10821 tablet_copy_client.cc:647] T 19ff9c216a094fde86ea9f9648bf08cb P d6ff17402670432da6eeb5da2af7b4b6: tablet copy: Starting download of 1 WAL segments...
I20240502 01:59:38.457146 10825 tablet_copy_client.cc:1002] T 19ff9c216a094fde86ea9f9648bf08cb P d6ff17402670432da6eeb5da2af7b4b6: tablet copy: Injecting latency into file download: 2000
I20240502 01:59:38.508612 10826 ts_tablet_manager.cc:902] T 4885222f238c4d8ea20ebc4c7e171694 P d52d70d7578f4995b03ac6faba3ec739: Initiating tablet copy from peer e77d1c0d41034d1eb5a036dc17c071f5 (127.6.117.195:43977)
I20240502 01:59:38.509227 10826 tablet_copy_client.cc:308] T 4885222f238c4d8ea20ebc4c7e171694 P d52d70d7578f4995b03ac6faba3ec739: tablet copy: Beginning tablet copy session from remote peer at address 127.6.117.195:43977
I20240502 01:59:38.513031 10218 tablet_copy_service.cc:140] P e77d1c0d41034d1eb5a036dc17c071f5: Received BeginTabletCopySession request for tablet 4885222f238c4d8ea20ebc4c7e171694 from peer d52d70d7578f4995b03ac6faba3ec739 ({username='slave'} at 127.0.0.1:48110)
I20240502 01:59:38.513134 10218 tablet_copy_service.cc:161] P e77d1c0d41034d1eb5a036dc17c071f5: Beginning new tablet copy session on tablet 4885222f238c4d8ea20ebc4c7e171694 from peer d52d70d7578f4995b03ac6faba3ec739 at {username='slave'} at 127.0.0.1:48110: session id = d52d70d7578f4995b03ac6faba3ec739-4885222f238c4d8ea20ebc4c7e171694
I20240502 01:59:38.514212 10218 tablet_copy_source_session.cc:205] T 4885222f238c4d8ea20ebc4c7e171694 P e77d1c0d41034d1eb5a036dc17c071f5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:59:38.515044 10826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4885222f238c4d8ea20ebc4c7e171694. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:38.517040 10826 tablet_copy_client.cc:783] T 4885222f238c4d8ea20ebc4c7e171694 P d52d70d7578f4995b03ac6faba3ec739: tablet copy: Starting download of 0 data blocks...
I20240502 01:59:38.517195 10826 tablet_copy_client.cc:647] T 4885222f238c4d8ea20ebc4c7e171694 P d52d70d7578f4995b03ac6faba3ec739: tablet copy: Starting download of 1 WAL segments...
I20240502 01:59:38.518254 10830 tablet_copy_client.cc:1002] T 4885222f238c4d8ea20ebc4c7e171694 P d52d70d7578f4995b03ac6faba3ec739: tablet copy: Injecting latency into file download: 2000
I20240502 01:59:38.522007 10833 ts_tablet_manager.cc:902] T 39f1842ff1494d78bfa0811a799c4bb8 P ae04d48c97204162b0f8a27d81fcec1a: Initiating tablet copy from peer bdbecaeb8f214529b2012a9c003de2ff (127.6.117.193:34597)
I20240502 01:59:38.522557 10833 tablet_copy_client.cc:308] T 39f1842ff1494d78bfa0811a799c4bb8 P ae04d48c97204162b0f8a27d81fcec1a: tablet copy: Beginning tablet copy session from remote peer at address 127.6.117.193:34597
I20240502 01:59:38.522970 10067 tablet_copy_service.cc:140] P bdbecaeb8f214529b2012a9c003de2ff: Received BeginTabletCopySession request for tablet 39f1842ff1494d78bfa0811a799c4bb8 from peer ae04d48c97204162b0f8a27d81fcec1a ({username='slave'} at 127.0.0.1:44462)
I20240502 01:59:38.523061 10067 tablet_copy_service.cc:161] P bdbecaeb8f214529b2012a9c003de2ff: Beginning new tablet copy session on tablet 39f1842ff1494d78bfa0811a799c4bb8 from peer ae04d48c97204162b0f8a27d81fcec1a at {username='slave'} at 127.0.0.1:44462: session id = ae04d48c97204162b0f8a27d81fcec1a-39f1842ff1494d78bfa0811a799c4bb8
I20240502 01:59:38.524139 10067 tablet_copy_source_session.cc:205] T 39f1842ff1494d78bfa0811a799c4bb8 P bdbecaeb8f214529b2012a9c003de2ff: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:59:38.525090 10833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39f1842ff1494d78bfa0811a799c4bb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:38.527639 10833 tablet_copy_client.cc:783] T 39f1842ff1494d78bfa0811a799c4bb8 P ae04d48c97204162b0f8a27d81fcec1a: tablet copy: Starting download of 0 data blocks...
I20240502 01:59:38.527783 10833 tablet_copy_client.cc:647] T 39f1842ff1494d78bfa0811a799c4bb8 P ae04d48c97204162b0f8a27d81fcec1a: tablet copy: Starting download of 1 WAL segments...
I20240502 01:59:38.528833 10837 tablet_copy_client.cc:1002] T 39f1842ff1494d78bfa0811a799c4bb8 P ae04d48c97204162b0f8a27d81fcec1a: tablet copy: Injecting latency into file download: 2000
I20240502 01:59:38.571136 10129 raft_consensus.cc:1261] T b4409766ce994f698c31aede604a6927 P 4b7f7ba2ac084e829c4304c16a94894f [term 4 FOLLOWER]: Refusing update from remote peer e77d1c0d41034d1eb5a036dc17c071f5: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 4 index: 30. (index mismatch)
I20240502 01:59:38.571646 10725 consensus_queue.cc:1035] T b4409766ce994f698c31aede604a6927 P e77d1c0d41034d1eb5a036dc17c071f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 01:59:38.573999 10590 raft_consensus.cc:1261] T b4409766ce994f698c31aede604a6927 P ae04d48c97204162b0f8a27d81fcec1a [term 4 FOLLOWER]: Refusing update from remote peer e77d1c0d41034d1eb5a036dc17c071f5: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 4 index: 30. (index mismatch)
I20240502 01:59:38.574657 10840 consensus_queue.cc:1035] T b4409766ce994f698c31aede604a6927 P e77d1c0d41034d1eb5a036dc17c071f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae04d48c97204162b0f8a27d81fcec1a" member_type: VOTER last_known_addr { host: "127.6.117.198" port: 33655 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 01:59:38.729735 10129 raft_consensus.cc:1261] T 73ea0bfb40ad43a68a6289ff1b35d005 P 4b7f7ba2ac084e829c4304c16a94894f [term 2 FOLLOWER]: Refusing update from remote peer d6ff17402670432da6eeb5da2af7b4b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240502 01:59:38.730096 10792 consensus_queue.cc:1035] T 73ea0bfb40ad43a68a6289ff1b35d005 P d6ff17402670432da6eeb5da2af7b4b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 01:59:38.733204 10053 raft_consensus.cc:1261] T 73ea0bfb40ad43a68a6289ff1b35d005 P bdbecaeb8f214529b2012a9c003de2ff [term 2 FOLLOWER]: Refusing update from remote peer d6ff17402670432da6eeb5da2af7b4b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240502 01:59:38.733847 10792 consensus_queue.cc:1035] T 73ea0bfb40ad43a68a6289ff1b35d005 P d6ff17402670432da6eeb5da2af7b4b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 01:59:38.740940 10129 raft_consensus.cc:1261] T 13d726dd71b14e84bc061f537f81e91d P 4b7f7ba2ac084e829c4304c16a94894f [term 3 FOLLOWER]: Refusing update from remote peer bdbecaeb8f214529b2012a9c003de2ff: Log matching property violated. Preceding OpId in replica: term: 2 index: 25. Preceding OpId from leader: term: 3 index: 26. (index mismatch)
I20240502 01:59:38.741264 10785 consensus_queue.cc:1035] T 13d726dd71b14e84bc061f537f81e91d P bdbecaeb8f214529b2012a9c003de2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 01:59:38.743841 10200 raft_consensus.cc:1261] T 13d726dd71b14e84bc061f537f81e91d P e77d1c0d41034d1eb5a036dc17c071f5 [term 3 FOLLOWER]: Refusing update from remote peer bdbecaeb8f214529b2012a9c003de2ff: Log matching property violated. Preceding OpId in replica: term: 2 index: 25. Preceding OpId from leader: term: 3 index: 26. (index mismatch)
I20240502 01:59:38.744197 10785 consensus_queue.cc:1035] T 13d726dd71b14e84bc061f537f81e91d P bdbecaeb8f214529b2012a9c003de2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 01:59:38.760304 10053 raft_consensus.cc:1261] T abf7d8b411bb46b68ac9ae77d74f8e2b P bdbecaeb8f214529b2012a9c003de2ff [term 2 FOLLOWER]: Refusing update from remote peer 4b7f7ba2ac084e829c4304c16a94894f: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240502 01:59:38.760711 10801 consensus_queue.cc:1035] T abf7d8b411bb46b68ac9ae77d74f8e2b P 4b7f7ba2ac084e829c4304c16a94894f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
W20240502 01:59:38.762869 10097 consensus_peers.cc:485] T abf7d8b411bb46b68ac9ae77d74f8e2b P 4b7f7ba2ac084e829c4304c16a94894f -> Peer ae04d48c97204162b0f8a27d81fcec1a (127.6.117.198:33655): Couldn't send request to peer ae04d48c97204162b0f8a27d81fcec1a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: abf7d8b411bb46b68ac9ae77d74f8e2b. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:38.763103 10202 raft_consensus.cc:1261] T abf7d8b411bb46b68ac9ae77d74f8e2b P e77d1c0d41034d1eb5a036dc17c071f5 [term 2 FOLLOWER]: Refusing update from remote peer 4b7f7ba2ac084e829c4304c16a94894f: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240502 01:59:38.763554 10731 consensus_queue.cc:1035] T abf7d8b411bb46b68ac9ae77d74f8e2b P 4b7f7ba2ac084e829c4304c16a94894f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e77d1c0d41034d1eb5a036dc17c071f5" member_type: VOTER last_known_addr { host: "127.6.117.195" port: 43977 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
W20240502 01:59:38.863371 10096 consensus_peers.cc:485] T 1851ea6467584b0ea32a1f68696803d0 P 4b7f7ba2ac084e829c4304c16a94894f -> Peer d52d70d7578f4995b03ac6faba3ec739 (127.6.117.197:44467): Couldn't send request to peer d52d70d7578f4995b03ac6faba3ec739. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1851ea6467584b0ea32a1f68696803d0. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:38.959321 10053 raft_consensus.cc:1261] T 1851ea6467584b0ea32a1f68696803d0 P bdbecaeb8f214529b2012a9c003de2ff [term 3 FOLLOWER]: Refusing update from remote peer 4b7f7ba2ac084e829c4304c16a94894f: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240502 01:59:38.959704 10731 consensus_queue.cc:1035] T 1851ea6467584b0ea32a1f68696803d0 P 4b7f7ba2ac084e829c4304c16a94894f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 01:59:38.961549 10797 raft_consensus.cc:979] T 1851ea6467584b0ea32a1f68696803d0 P 4b7f7ba2ac084e829c4304c16a94894f: : Instructing follower bdbecaeb8f214529b2012a9c003de2ff to start an election
I20240502 01:59:38.961650 10801 raft_consensus.cc:1067] T 1851ea6467584b0ea32a1f68696803d0 P 4b7f7ba2ac084e829c4304c16a94894f [term 3 LEADER]: Signalling peer bdbecaeb8f214529b2012a9c003de2ff to start an election
I20240502 01:59:38.962097 10054 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1851ea6467584b0ea32a1f68696803d0"
dest_uuid: "bdbecaeb8f214529b2012a9c003de2ff"
from {username='slave'} at 127.0.0.1:44396
I20240502 01:59:38.962221 10054 raft_consensus.cc:482] T 1851ea6467584b0ea32a1f68696803d0 P bdbecaeb8f214529b2012a9c003de2ff [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:59:38.962275 10054 raft_consensus.cc:3037] T 1851ea6467584b0ea32a1f68696803d0 P bdbecaeb8f214529b2012a9c003de2ff [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:59:38.962663 10441 raft_consensus.cc:1261] T 1851ea6467584b0ea32a1f68696803d0 P d6ff17402670432da6eeb5da2af7b4b6 [term 3 FOLLOWER]: Refusing update from remote peer 4b7f7ba2ac084e829c4304c16a94894f: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240502 01:59:38.963021 10731 consensus_queue.cc:1035] T 1851ea6467584b0ea32a1f68696803d0 P 4b7f7ba2ac084e829c4304c16a94894f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6ff17402670432da6eeb5da2af7b4b6" member_type: VOTER last_known_addr { host: "127.6.117.196" port: 34337 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 01:59:38.963151 10054 raft_consensus.cc:504] T 1851ea6467584b0ea32a1f68696803d0 P bdbecaeb8f214529b2012a9c003de2ff [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } attrs { replace: true } } peers { permanent_uuid: "d6ff17402670432da6eeb5da2af7b4b6" member_type: VOTER last_known_addr { host: "127.6.117.196" port: 34337 } attrs { promote: false } } peers { permanent_uuid: "d52d70d7578f4995b03ac6faba3ec739" member_type: NON_VOTER last_known_addr { host: "127.6.117.197" port: 44467 } attrs { promote: true } }
I20240502 01:59:38.963325 10053 raft_consensus.cc:1226] T 1851ea6467584b0ea32a1f68696803d0 P bdbecaeb8f214529b2012a9c003de2ff [term 4 FOLLOWER]: Rejecting Update request from peer 4b7f7ba2ac084e829c4304c16a94894f for earlier term 3. Current term is 4. Ops: []
I20240502 01:59:38.963485 10054 leader_election.cc:288] T 1851ea6467584b0ea32a1f68696803d0 P bdbecaeb8f214529b2012a9c003de2ff [CANDIDATE]: Term 4 election: Requested vote from peers 4b7f7ba2ac084e829c4304c16a94894f (127.6.117.194:46805), d6ff17402670432da6eeb5da2af7b4b6 (127.6.117.196:34337)
I20240502 01:59:38.963635 10731 consensus_queue.cc:1046] T 1851ea6467584b0ea32a1f68696803d0 P 4b7f7ba2ac084e829c4304c16a94894f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 }, Status: INVALID_TERM, Last received: 3.31, Next index: 32, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 01:59:38.963780 10797 raft_consensus.cc:3032] T 1851ea6467584b0ea32a1f68696803d0 P 4b7f7ba2ac084e829c4304c16a94894f [term 3 LEADER]: Stepping down as leader of term 3
I20240502 01:59:38.963836 10797 raft_consensus.cc:729] T 1851ea6467584b0ea32a1f68696803d0 P 4b7f7ba2ac084e829c4304c16a94894f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4b7f7ba2ac084e829c4304c16a94894f, State: Running, Role: LEADER
I20240502 01:59:38.963968 10797 consensus_queue.cc:260] T 1851ea6467584b0ea32a1f68696803d0 P 4b7f7ba2ac084e829c4304c16a94894f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 3.31, Last appended by leader: 31, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } attrs { replace: true } } peers { permanent_uuid: "d6ff17402670432da6eeb5da2af7b4b6" member_type: VOTER last_known_addr { host: "127.6.117.196" port: 34337 } attrs { promote: false } } peers { permanent_uuid: "d52d70d7578f4995b03ac6faba3ec739" member_type: NON_VOTER last_known_addr { host: "127.6.117.197" port: 44467 } attrs { promote: true } }
I20240502 01:59:38.964144 10797 raft_consensus.cc:3037] T 1851ea6467584b0ea32a1f68696803d0 P 4b7f7ba2ac084e829c4304c16a94894f [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:59:38.964128 10440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1851ea6467584b0ea32a1f68696803d0" candidate_uuid: "bdbecaeb8f214529b2012a9c003de2ff" candidate_term: 4 candidate_status { last_received { term: 3 index: 31 } } ignore_live_leader: true dest_uuid: "d6ff17402670432da6eeb5da2af7b4b6"
I20240502 01:59:38.964891 10440 raft_consensus.cc:3037] T 1851ea6467584b0ea32a1f68696803d0 P d6ff17402670432da6eeb5da2af7b4b6 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:59:38.965813 10440 raft_consensus.cc:2454] T 1851ea6467584b0ea32a1f68696803d0 P d6ff17402670432da6eeb5da2af7b4b6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdbecaeb8f214529b2012a9c003de2ff in term 4.
I20240502 01:59:38.966246 10021 leader_election.cc:302] T 1851ea6467584b0ea32a1f68696803d0 P bdbecaeb8f214529b2012a9c003de2ff [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bdbecaeb8f214529b2012a9c003de2ff, d6ff17402670432da6eeb5da2af7b4b6; no voters:
I20240502 01:59:38.966410 10785 raft_consensus.cc:2781] T 1851ea6467584b0ea32a1f68696803d0 P bdbecaeb8f214529b2012a9c003de2ff [term 4 FOLLOWER]: Leader election won for term 4
I20240502 01:59:38.966498 10785 raft_consensus.cc:686] T 1851ea6467584b0ea32a1f68696803d0 P bdbecaeb8f214529b2012a9c003de2ff [term 4 LEADER]: Becoming Leader. State: Replica: bdbecaeb8f214529b2012a9c003de2ff, State: Running, Role: LEADER
I20240502 01:59:38.966615 10785 consensus_queue.cc:237] T 1851ea6467584b0ea32a1f68696803d0 P bdbecaeb8f214529b2012a9c003de2ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.31, Last appended by leader: 31, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } attrs { replace: true } } peers { permanent_uuid: "d6ff17402670432da6eeb5da2af7b4b6" member_type: VOTER last_known_addr { host: "127.6.117.196" port: 34337 } attrs { promote: false } } peers { permanent_uuid: "d52d70d7578f4995b03ac6faba3ec739" member_type: NON_VOTER last_known_addr { host: "127.6.117.197" port: 44467 } attrs { promote: true } }
I20240502 01:59:38.966147 10131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1851ea6467584b0ea32a1f68696803d0" candidate_uuid: "bdbecaeb8f214529b2012a9c003de2ff" candidate_term: 4 candidate_status { last_received { term: 3 index: 31 } } ignore_live_leader: true dest_uuid: "4b7f7ba2ac084e829c4304c16a94894f"
I20240502 01:59:38.967813 10131 raft_consensus.cc:2454] T 1851ea6467584b0ea32a1f68696803d0 P 4b7f7ba2ac084e829c4304c16a94894f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdbecaeb8f214529b2012a9c003de2ff in term 4.
I20240502 01:59:38.968099 9948 catalog_manager.cc:5484] T 1851ea6467584b0ea32a1f68696803d0 P bdbecaeb8f214529b2012a9c003de2ff reported cstate change: term changed from 3 to 4, leader changed from 4b7f7ba2ac084e829c4304c16a94894f (127.6.117.194) to bdbecaeb8f214529b2012a9c003de2ff (127.6.117.193). New cstate: current_term: 4 leader_uuid: "bdbecaeb8f214529b2012a9c003de2ff" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "bdbecaeb8f214529b2012a9c003de2ff" member_type: VOTER last_known_addr { host: "127.6.117.193" port: 34597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4b7f7ba2ac084e829c4304c16a94894f" member_type: VOTER last_known_addr { host: "127.6.117.194" port: 46805 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6ff17402670432da6eeb5da2af7b4b6" member_type: VOTER last_known_addr { host: "127.6.117.196" port: 34337 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d52d70d7578f4995b03ac6faba3ec739" member_type: NON_VOTER last_known_addr { host: "127.6.117.197" port: 44467 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:59:39.318822 10859 ts_tablet_manager.cc:902] T abf7d8b411bb46b68ac9ae77d74f8e2b P ae04d48c97204162b0f8a27d81fcec1a: Initiating tablet copy from peer 4b7f7ba2ac084e829c4304c16a94894f (127.6.117.194:46805)
F20240502 01:59:39.319103 10859 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (init.queue_time_us, init.queue_time_us) (init.run_cpu_time_us, init.run_cpu_time_us) (init.run_wall_time_us, init.run_wall_time_us) (leader-initialization.queue_time_us, leader-initialization.queue_time_us) (leader-initialization.run_cpu_time_us, leader-initialization.run_cpu_time_us) (leader-initialization.run_wall_time_us, leader-initialization.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-01d518fafee34875a7df792d8297834b.queue_time_us, tablet-download-pool-01d518fafee34875a7df792d8297834b.queue_time_us) (tablet-download-pool-01d518fafee34875a7df792d8297834b.run_cpu_time_us, tablet-download-pool-01d518fafee34875a7df792d8297834b.run_cpu_time_us) (tablet-download-pool-01d518fafee34875a7df792d8297834b.run_wall_time_us, tablet-download-pool-01d518fafee34875a7df792d8297834b.run_wall_time_us) (tablet-download-pool-1851ea6467584b0ea32a1f68696803d0.queue_time_us, tablet-download-pool-1851ea6467584b0ea32a1f68696803d0.queue_time_us) (tablet-download-pool-1851ea6467584b0ea32a1f68696803d0.run_cpu_time_us, tablet-download-pool-1851ea6467584b0ea32a1f68696803d0.run_cpu_time_us) (tablet-download-pool-1851ea6467584b0ea32a1f68696803d0.run_wall_time_us, tablet-download-pool-1851ea6467584b0ea32a1f68696803d0.run_wall_time_us) (tablet-download-pool-19ff9c216a094fde86ea9f9648bf08cb.queue_time_us, tablet-download-pool-19ff9c216a094fde86ea9f9648bf08cb.queue_time_us) (tablet-download-pool-19ff9c216a094fde86ea9f9648bf08cb.run_cpu_time_us, tablet-download-pool-19ff9c216a094fde86ea9f9648bf08cb.run_cpu_time_us) (tablet-download-pool-19ff9c216a094fde86ea9f9648bf08cb.run_wall_time_us, tablet-download-pool-19ff9c216a094fde86ea9f9648bf08cb.run_wall_time_us) (tablet-download-pool-39f1842ff1494d78bfa0811a799c4bb8.queue_time_us, tablet-download-pool-39f1842ff1494d78bfa0811a799c4bb8.queue_time_us) (tablet-download-pool-39f1842ff1494d78bfa0811a799c4bb8.run_cpu_time_us, tablet-download-pool-39f1842ff1494d78bfa0811a799c4bb8.run_cpu_time_us) (tablet-download-pool-39f1842ff1494d78bfa0811a799c4bb8.run_wall_time_us, tablet-download-pool-39f1842ff1494d78bfa0811a799c4bb8.run_wall_time_us) (tablet-download-pool-42d98987ce3e4e82ad0e55e00cd49c24.queue_time_us, tablet-download-pool-42d98987ce3e4e82ad0e55e00cd49c24.queue_time_us) (tablet-download-pool-42d98987ce3e4e82ad0e55e00cd49c24.run_cpu_time_us, tablet-download-pool-42d98987ce3e4e82ad0e55e00cd49c24.run_cpu_time_us) (tablet-download-pool-42d98987ce3e4e82ad0e55e00cd49c24.run_wall_time_us, tablet-download-pool-42d98987ce3e4e82ad0e55e00cd49c24.run_wall_time_us) (tablet-download-pool-4885222f238c4d8ea20ebc4c7e171694.queue_time_us, tablet-download-pool-4885222f238c4d8ea20ebc4c7e171694.queue_time_us) (tablet-download-pool-4885222f238c4d8ea20ebc4c7e171694.run_cpu_time_us, tablet-download-pool-4885222f238c4d8ea20ebc4c7e171694.run_cpu_time_us) (tablet-download-pool-4885222f238c4d8ea20ebc4c7e171694.run_wall_time_us, tablet-download-pool-4885222f238c4d8ea20ebc4c7e171694.run_wall_time_us) (tablet-download-pool-580db11a61cb46f48d5f7507cfbdbb52.queue_time_us, tablet-download-pool-580db11a61cb46f48d5f7507cfbdbb52.queue_time_us) (tablet-download-pool-580db11a61cb46f48d5f7507cfbdbb52.run_cpu_time_us, tablet-download-pool-580db11a61cb46f48d5f7507cfbdbb52.run_cpu_time_us) (tablet-download-pool-580db11a61cb46f48d5f7507cfbdbb52.run_wall_time_us, tablet-download-pool-580db11a61cb46f48d5f7507cfbdbb52.run_wall_time_us) (tablet-download-pool-73ea0bfb40ad43a68a6289ff1b35d005.queue_time_us, tablet-download-pool-73ea0bfb40ad43a68a6289ff1b35d005.queue_time_us) (tablet-download-pool-73ea0bfb40ad43a68a6289ff1b35d005.run_cpu_time_us, tablet-download-pool-73ea0bfb40ad43a68a6289ff1b35d005.run_cpu_time_us) (tablet-download-pool-73ea0bfb40ad43a68a6289ff1b35d005.run_wall_time_us, tablet-download-pool-73ea0bfb40ad43a68a6289ff1b35d005.run_wall_time_us) (tablet-download-pool-abf7d8b411bb46b68ac9ae77d74f8e2b.queue_time_us, tablet-download-pool-abf7d8b411bb46b68ac9ae77d74f8e2b.queue_time_us) (tablet-download-pool-b404009b1e4e4be89b787476201c68a9.queue_time_us, tablet-download-pool-b404009b1e4e4be89b787476201c68a9.queue_time_us) (tablet-download-pool-b404009b1e4e4be89b787476201c68a9.run_cpu_time_us, tablet-download-pool-b404009b1e4e4be89b787476201c68a9.run_cpu_time_us) (tablet-download-pool-b404009b1e4e4be89b787476201c68a9.run_wall_time_us, tablet-download-pool-b404009b1e4e4be89b787476201c68a9.run_wall_time_us) (tablet-download-pool-b4409766ce994f698c31aede604a6927.queue_time_us, tablet-download-pool-b4409766ce994f698c31aede604a6927.queue_time_us) (tablet-download-pool-b4409766ce994f698c31aede604a6927.run_cpu_time_us, tablet-download-pool-b4409766ce994f698c31aede604a6927.run_cpu_time_us) (tablet-download-pool-b4409766ce994f698c31aede604a6927.run_wall_time_us, tablet-download-pool-b4409766ce994f698c31aede604a6927.run_wall_time_us) (tablet-download-pool-e5668a318ea44db8a7c4db7fc7c95f7a.queue_time_us, tablet-download-pool-e5668a318ea44db8a7c4db7fc7c95f7a.queue_time_us) (tablet-download-pool-e5668a318ea44db8a7c4db7fc7c95f7a.run_cpu_time_us, tablet-download-pool-e5668a318ea44db8a7c4db7fc7c95f7a.run_cpu_time_us) (tablet-download-pool-e5668a318ea44db8a7c4db7fc7c95f7a.run_wall_time_us, tablet-download-pool-e5668a318ea44db8a7c4db7fc7c95f7a.run_wall_time_us) (tablet-download-pool-e7a8fc5afcab4b1db4ebf245b66df4be.queue_time_us, tablet-download-pool-e7a8fc5afcab4b1db4ebf245b66df4be.queue_time_us) (tablet-download-pool-e7a8fc5afcab4b1db4ebf245b66df4be.run_cpu_time_us, tablet-download-pool-e7a8fc5afcab4b1db4ebf245b66df4be.run_cpu_time_us) (tablet-download-pool-e7a8fc5afcab4b1db4ebf245b66df4be.run_wall_time_us, tablet-download-pool-e7a8fc5afcab4b1db4ebf245b66df4be.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
*** Aborted at 1714615179 (unix time) try "date -d @1714615179" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e8000019d7) received by PID 6615 (TID 0x7f6072cf1700) from PID 6615; stack trace: ***
@ 0x7f613a7c6040 (unknown) at ??:0
@ 0x7f613a7c5fb7 gsignal at ??:0
@ 0x7f613a7c7921 abort at ??:0
@ 0x7f613b4cadcd google::LogMessage::Fail() at ??:0
@ 0x7f613b4ceb93 google::LogMessage::SendToLog() at ??:0
@ 0x7f613b4ca7cc google::LogMessage::Flush() at ??:0
@ 0x7f613b4cbf59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f613c022044 kudu::TraceMetrics::InternName() at ??:0
@ 0x7f613c006f98 kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f613c004efc kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f613cf8177a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f613cf8aa45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7f613d00b76f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7f613d01f4f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f613d00a226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
@ 0x7f613d019c0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f613d5743cc std::function<>::operator()() at ??:0
@ 0x7f613c009e06 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f613c00a6ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f613c00c03b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f613d5743cc std::function<>::operator()() at ??:0
@ 0x7f613bff9807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f61387fb6db start_thread at ??:0
@ 0x7f613a8a871f clone at ??:0