[==========] Running 14 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 14 tests from AutoRebalancerTest
[ RUN ] AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240503 14:11:55.312969 16762 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.16.94.190:41163,127.16.94.189:39523,127.16.94.188:42477
I20240503 14:11:55.314738 16762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:11:55.315824 16762 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:11:55.326503 16767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:11:55.327085 16770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:11:55.327726 16768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:11:55.328451 16762 server_base.cc:1034] running on GCE node
I20240503 14:11:55.329214 16762 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:11:55.329381 16762 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:11:55.329465 16762 hybrid_clock.cc:648] HybridClock initialized: now 1714745515329443 us; error 0 us; skew 500 ppm
I20240503 14:11:55.329829 16762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:11:55.332440 16762 webserver.cc:460] Webserver started at http://127.16.94.190:34699/ using document root <none> and password file <none>
I20240503 14:11:55.333179 16762 fs_manager.cc:359] Metadata directory not provided
I20240503 14:11:55.333285 16762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:11:55.333622 16762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:11:55.336971 16762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714745515241994-16762-0/minicluster-data/master-0-root/instance:
uuid: "9225ffbbab1a44839573abeb7b35f298"
format_stamp: "Formatted at 2024-05-03 14:11:55 on dist-test-slave-v27g"
I20240503 14:11:55.342718 16762 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.006s
I20240503 14:11:55.346006 16775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:55.347391 16762 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240503 14:11:55.347563 16762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714745515241994-16762-0/minicluster-data/master-0-root
uuid: "9225ffbbab1a44839573abeb7b35f298"
format_stamp: "Formatted at 2024-05-03 14:11:55 on dist-test-slave-v27g"
I20240503 14:11:55.347743 16762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714745515241994-16762-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714745515241994-16762-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714745515241994-16762-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:11:55.377090 16762 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:11:55.377938 16762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:11:55.378221 16762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:11:55.396229 16762 rpc_server.cc:307] RPC server started. Bound to: 127.16.94.190:41163
I20240503 14:11:55.396271 16826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.94.190:41163 every 8 connection(s)
I20240503 14:11:55.398375 16762 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 14:11:55.401504 16827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:11:55.401888 16830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:11:55.402338 16829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:11:55.403100 16832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:11:55.403610 16762 server_base.cc:1034] running on GCE node
I20240503 14:11:55.403931 16762 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:11:55.404019 16762 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:11:55.404093 16762 hybrid_clock.cc:648] HybridClock initialized: now 1714745515404056 us; error 0 us; skew 500 ppm
I20240503 14:11:55.404321 16762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:11:55.405866 16762 webserver.cc:460] Webserver started at http://127.16.94.189:34709/ using document root <none> and password file <none>
I20240503 14:11:55.406153 16762 fs_manager.cc:359] Metadata directory not provided
I20240503 14:11:55.406260 16762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:11:55.406405 16762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:11:55.407222 16762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714745515241994-16762-0/minicluster-data/master-1-root/instance:
uuid: "173b293a9078456ea4235d5503519f83"
format_stamp: "Formatted at 2024-05-03 14:11:55 on dist-test-slave-v27g"
I20240503 14:11:55.410605 16762 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20240503 14:11:55.412760 16837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:55.413204 16762 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.002s
I20240503 14:11:55.413314 16762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714745515241994-16762-0/minicluster-data/master-1-root
uuid: "173b293a9078456ea4235d5503519f83"
format_stamp: "Formatted at 2024-05-03 14:11:55 on dist-test-slave-v27g"
I20240503 14:11:55.413450 16762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714745515241994-16762-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714745515241994-16762-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714745515241994-16762-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:11:55.415638 16827 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.94.190" port: 41163 } has no permanent_uuid. Determining permanent_uuid...
I20240503 14:11:55.425765 16762 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:11:55.426621 16762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:11:55.432538 16827 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.94.189" port: 39523 } has no permanent_uuid. Determining permanent_uuid...
W20240503 14:11:55.433987 16777 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.94.189:39523: connect: Connection refused (error 111)
W20240503 14:11:55.436051 16827 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.16.94.189:39523: Network error: Client connection negotiation failed: client connection to 127.16.94.189:39523: connect: Connection refused (error 111)
I20240503 14:11:55.443176 16762 rpc_server.cc:307] RPC server started. Bound to: 127.16.94.189:39523
I20240503 14:11:55.443210 16892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.94.189:39523 every 8 connection(s)
I20240503 14:11:55.444463 16762 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 14:11:55.444742 16893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:11:55.448498 16896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:11:55.448969 16893 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.94.190" port: 41163 } has no permanent_uuid. Determining permanent_uuid...
W20240503 14:11:55.449128 16895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:11:55.450047 16762 server_base.cc:1034] running on GCE node
W20240503 14:11:55.450214 16898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:11:55.450614 16762 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:11:55.450704 16762 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:11:55.450845 16762 hybrid_clock.cc:648] HybridClock initialized: now 1714745515450801 us; error 0 us; skew 500 ppm
I20240503 14:11:55.451050 16762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:11:55.452577 16762 webserver.cc:460] Webserver started at http://127.16.94.188:38803/ using document root <none> and password file <none>
I20240503 14:11:55.452822 16762 fs_manager.cc:359] Metadata directory not provided
I20240503 14:11:55.452927 16762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:11:55.453075 16762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:11:55.453768 16762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714745515241994-16762-0/minicluster-data/master-2-root/instance:
uuid: "fc675fa1ed884b8bac510acadd18cd79"
format_stamp: "Formatted at 2024-05-03 14:11:55 on dist-test-slave-v27g"
I20240503 14:11:55.456255 16893 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.94.189" port: 39523 } has no permanent_uuid. Determining permanent_uuid...
I20240503 14:11:55.456840 16762 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20240503 14:11:55.459050 16905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:55.459429 16762 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 14:11:55.459551 16762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714745515241994-16762-0/minicluster-data/master-2-root
uuid: "fc675fa1ed884b8bac510acadd18cd79"
format_stamp: "Formatted at 2024-05-03 14:11:55 on dist-test-slave-v27g"
I20240503 14:11:55.459687 16762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714745515241994-16762-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714745515241994-16762-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714745515241994-16762-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:11:55.463909 16893 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.94.188" port: 42477 } has no permanent_uuid. Determining permanent_uuid...
W20240503 14:11:55.466082 16893 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.16.94.188:42477: Network error: Client connection negotiation failed: client connection to 127.16.94.188:42477: connect: Connection refused (error 111)
I20240503 14:11:55.480311 16762 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:11:55.480885 16762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:11:55.485423 16827 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.94.189" port: 39523 } attempt: 1
I20240503 14:11:55.494485 16827 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.94.188" port: 42477 } has no permanent_uuid. Determining permanent_uuid...
W20240503 14:11:55.496731 16827 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.16.94.188:42477: Network error: Client connection negotiation failed: client connection to 127.16.94.188:42477: connect: Connection refused (error 111)
I20240503 14:11:55.498916 16762 rpc_server.cc:307] RPC server started. Bound to: 127.16.94.188:42477
I20240503 14:11:55.499006 16957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.94.188:42477 every 8 connection(s)
I20240503 14:11:55.499958 16762 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 14:11:55.500523 16959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:11:55.502952 16959 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.94.190" port: 41163 } has no permanent_uuid. Determining permanent_uuid...
I20240503 14:11:55.508991 16959 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.94.189" port: 39523 } has no permanent_uuid. Determining permanent_uuid...
I20240503 14:11:55.514371 16959 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.94.188" port: 42477 } has no permanent_uuid. Determining permanent_uuid...
I20240503 14:11:55.518271 16893 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.94.188" port: 42477 } attempt: 1
I20240503 14:11:55.525597 16959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc675fa1ed884b8bac510acadd18cd79: Bootstrap starting.
I20240503 14:11:55.527290 16893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 173b293a9078456ea4235d5503519f83: Bootstrap starting.
I20240503 14:11:55.529266 16959 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc675fa1ed884b8bac510acadd18cd79: Neither blocks nor log segments found. Creating new log.
I20240503 14:11:55.529269 16893 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 173b293a9078456ea4235d5503519f83: Neither blocks nor log segments found. Creating new log.
I20240503 14:11:55.530706 16893 log.cc:824] T 00000000000000000000000000000000 P 173b293a9078456ea4235d5503519f83: Log is configured to *not* fsync() on all Append() calls
I20240503 14:11:55.533200 16893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 173b293a9078456ea4235d5503519f83: No bootstrap required, opened a new log
I20240503 14:11:55.533210 16959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc675fa1ed884b8bac510acadd18cd79: No bootstrap required, opened a new log
I20240503 14:11:55.539556 16959 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc675fa1ed884b8bac510acadd18cd79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9225ffbbab1a44839573abeb7b35f298" member_type: VOTER last_known_addr { host: "127.16.94.190" port: 41163 } } peers { permanent_uuid: "173b293a9078456ea4235d5503519f83" member_type: VOTER last_known_addr { host: "127.16.94.189" port: 39523 } } peers { permanent_uuid: "fc675fa1ed884b8bac510acadd18cd79" member_type: VOTER last_known_addr { host: "127.16.94.188" port: 42477 } }
I20240503 14:11:55.539562 16893 raft_consensus.cc:348] T 00000000000000000000000000000000 P 173b293a9078456ea4235d5503519f83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9225ffbbab1a44839573abeb7b35f298" member_type: VOTER last_known_addr { host: "127.16.94.190" port: 41163 } } peers { permanent_uuid: "173b293a9078456ea4235d5503519f83" member_type: VOTER last_known_addr { host: "127.16.94.189" port: 39523 } } peers { permanent_uuid: "fc675fa1ed884b8bac510acadd18cd79" member_type: VOTER last_known_addr { host: "127.16.94.188" port: 42477 } }
I20240503 14:11:55.539866 16893 raft_consensus.cc:374] T 00000000000000000000000000000000 P 173b293a9078456ea4235d5503519f83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:11:55.539876 16959 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc675fa1ed884b8bac510acadd18cd79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:11:55.539934 16827 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.94.188" port: 42477 } attempt: 1
I20240503 14:11:55.540040 16959 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc675fa1ed884b8bac510acadd18cd79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc675fa1ed884b8bac510acadd18cd79, State: Initialized, Role: FOLLOWER
I20240503 14:11:55.540045 16893 raft_consensus.cc:729] T 00000000000000000000000000000000 P 173b293a9078456ea4235d5503519f83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 173b293a9078456ea4235d5503519f83, State: Initialized, Role: FOLLOWER
I20240503 14:11:55.540887 16893 consensus_queue.cc:260] T 00000000000000000000000000000000 P 173b293a9078456ea4235d5503519f83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9225ffbbab1a44839573abeb7b35f298" member_type: VOTER last_known_addr { host: "127.16.94.190" port: 41163 } } peers { permanent_uuid: "173b293a9078456ea4235d5503519f83" member_type: VOTER last_known_addr { host: "127.16.94.189" port: 39523 } } peers { permanent_uuid: "fc675fa1ed884b8bac510acadd18cd79" member_type: VOTER last_known_addr { host: "127.16.94.188" port: 42477 } }
I20240503 14:11:55.540889 16959 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc675fa1ed884b8bac510acadd18cd79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9225ffbbab1a44839573abeb7b35f298" member_type: VOTER last_known_addr { host: "127.16.94.190" port: 41163 } } peers { permanent_uuid: "173b293a9078456ea4235d5503519f83" member_type: VOTER last_known_addr { host: "127.16.94.189" port: 39523 } } peers { permanent_uuid: "fc675fa1ed884b8bac510acadd18cd79" member_type: VOTER last_known_addr { host: "127.16.94.188" port: 42477 } }
I20240503 14:11:55.542222 16966 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc675fa1ed884b8bac510acadd18cd79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9225ffbbab1a44839573abeb7b35f298" member_type: VOTER last_known_addr { host: "127.16.94.190" port: 41163 } } peers { permanent_uuid: "173b293a9078456ea4235d5503519f83" member_type: VOTER last_known_addr { host: "127.16.94.189" port: 39523 } } peers { permanent_uuid: "fc675fa1ed884b8bac510acadd18cd79" member_type: VOTER last_known_addr { host: "127.16.94.188" port: 42477 } } }
I20240503 14:11:55.543396 16966 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc675fa1ed884b8bac510acadd18cd79 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 14:11:55.543128 16967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 173b293a9078456ea4235d5503519f83 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9225ffbbab1a44839573abeb7b35f298" member_type: VOTER last_known_addr { host: "127.16.94.190" port: 41163 } } peers { permanent_uuid: "173b293a9078456ea4235d5503519f83" member_type: VOTER last_known_addr { host: "127.16.94.189" port: 39523 } } peers { permanent_uuid: "fc675fa1ed884b8bac510acadd18cd79" member_type: VOTER last_known_addr { host: "127.16.94.188" port: 42477 } } }
I20240503 14:11:55.543586 16959 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc675fa1ed884b8bac510acadd18cd79 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 14:11:55.543593 16893 sys_catalog.cc:564] T 00000000000000000000000000000000 P 173b293a9078456ea4235d5503519f83 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 14:11:55.543663 16967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 173b293a9078456ea4235d5503519f83 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 14:11:55.552549 16827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9225ffbbab1a44839573abeb7b35f298: Bootstrap starting.
I20240503 14:11:55.554746 16827 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9225ffbbab1a44839573abeb7b35f298: Neither blocks nor log segments found. Creating new log.
W20240503 14:11:55.558801 16989 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fc675fa1ed884b8bac510acadd18cd79: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 14:11:55.558990 16989 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20240503 14:11:55.558827 16987 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 173b293a9078456ea4235d5503519f83: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 14:11:55.559171 16987 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240503 14:11:55.560034 16827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9225ffbbab1a44839573abeb7b35f298: No bootstrap required, opened a new log
I20240503 14:11:55.560742 16827 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9225ffbbab1a44839573abeb7b35f298 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9225ffbbab1a44839573abeb7b35f298" member_type: VOTER last_known_addr { host: "127.16.94.190" port: 41163 } } peers { permanent_uuid: "173b293a9078456ea4235d5503519f83" member_type: VOTER last_known_addr { host: "127.16.94.189" port: 39523 } } peers { permanent_uuid: "fc675fa1ed884b8bac510acadd18cd79" member_type: VOTER last_known_addr { host: "127.16.94.188" port: 42477 } }
I20240503 14:11:55.560964 16827 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9225ffbbab1a44839573abeb7b35f298 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:11:55.561053 16827 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9225ffbbab1a44839573abeb7b35f298 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9225ffbbab1a44839573abeb7b35f298, State: Initialized, Role: FOLLOWER
I20240503 14:11:55.561309 16827 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9225ffbbab1a44839573abeb7b35f298 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9225ffbbab1a44839573abeb7b35f298" member_type: VOTER last_known_addr { host: "127.16.94.190" port: 41163 } } peers { permanent_uuid: "173b293a9078456ea4235d5503519f83" member_type: VOTER last_known_addr { host: "127.16.94.189" port: 39523 } } peers { permanent_uuid: "fc675fa1ed884b8bac510acadd18cd79" member_type: VOTER last_known_addr { host: "127.16.94.188" port: 42477 } }
I20240503 14:11:55.562062 16992 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9225ffbbab1a44839573abeb7b35f298 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9225ffbbab1a44839573abeb7b35f298" member_type: VOTER last_known_addr { host: "127.16.94.190" port: 41163 } } peers { permanent_uuid: "173b293a9078456ea4235d5503519f83" member_type: VOTER last_known_addr { host: "127.16.94.189" port: 39523 } } peers { permanent_uuid: "fc675fa1ed884b8bac510acadd18cd79" member_type: VOTER last_known_addr { host: "127.16.94.188" port: 42477 } } }
I20240503 14:11:55.562330 16992 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9225ffbbab1a44839573abeb7b35f298 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 14:11:55.562386 16827 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9225ffbbab1a44839573abeb7b35f298 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 14:11:55.567359 16762 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
I20240503 14:11:55.567479 16762 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
I20240503 14:11:55.568498 16762 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:11:55.568653 17003 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9225ffbbab1a44839573abeb7b35f298: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 14:11:55.568755 17003 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20240503 14:11:55.573205 17004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:11:55.573880 17005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:11:55.574582 16762 server_base.cc:1034] running on GCE node
W20240503 14:11:55.575126 17007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:11:55.575526 16762 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:11:55.575650 16762 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:11:55.575716 16762 hybrid_clock.cc:648] HybridClock initialized: now 1714745515575695 us; error 0 us; skew 500 ppm
I20240503 14:11:55.575942 16762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:11:55.577375 16762 webserver.cc:460] Webserver started at http://127.16.94.129:45165/ using document root <none> and password file <none>
I20240503 14:11:55.577714 16762 fs_manager.cc:359] Metadata directory not provided
I20240503 14:11:55.577818 16762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:11:55.577986 16762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:11:55.578898 16762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714745515241994-16762-0/minicluster-data/ts-0-root/instance:
uuid: "df5aa9fb874045759a08adf79fd4f0f7"
format_stamp: "Formatted at 2024-05-03 14:11:55 on dist-test-slave-v27g"
I20240503 14:11:55.582525 16762 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20240503 14:11:55.584659 17012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:55.585055 16762 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240503 14:11:55.585188 16762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714745515241994-16762-0/minicluster-data/ts-0-root
uuid: "df5aa9fb874045759a08adf79fd4f0f7"
format_stamp: "Formatted at 2024-05-03 14:11:55 on dist-test-slave-v27g"
I20240503 14:11:55.585335 16762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714745515241994-16762-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714745515241994-16762-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714745515241994-16762-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:11:55.598932 16762 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:11:55.599671 16762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:11:55.600628 16762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:11:55.602547 16762 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:11:55.602690 16762 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:55.602803 16762 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:11:55.602895 16762 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:55.622413 16762 rpc_server.cc:307] RPC server started. Bound to: 127.16.94.129:37917
I20240503 14:11:55.622424 17074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.94.129:37917 every 8 connection(s)
W20240503 14:11:55.630249 16781 tablet.cc:2369] T 00000000000000000000000000000000 P 9225ffbbab1a44839573abeb7b35f298: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:11:55.645036 16762 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 14:11:55.646725 17076 heartbeater.cc:347] Connected to a master server at 127.16.94.190:41163
I20240503 14:11:55.647087 17076 heartbeater.cc:464] Registering TS with master...
I20240503 14:11:55.648234 17076 heartbeater.cc:511] Master 127.16.94.190:41163 requested a full tablet report, sending...
I20240503 14:11:55.651120 16792 ts_manager.cc:194] Registered new tserver with Master: df5aa9fb874045759a08adf79fd4f0f7 (127.16.94.129:37917)
W20240503 14:11:55.656430 17082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:11:55.656702 17075 heartbeater.cc:347] Connected to a master server at 127.16.94.188:42477
I20240503 14:11:55.656839 17075 heartbeater.cc:464] Registering TS with master...
I20240503 14:11:55.657249 17075 heartbeater.cc:511] Master 127.16.94.188:42477 requested a full tablet report, sending...
I20240503 14:11:55.658551 16923 ts_manager.cc:194] Registered new tserver with Master: df5aa9fb874045759a08adf79fd4f0f7 (127.16.94.129:37917)
W20240503 14:11:55.659189 17083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:11:55.660625 17077 heartbeater.cc:347] Connected to a master server at 127.16.94.189:39523
I20240503 14:11:55.660800 17077 heartbeater.cc:464] Registering TS with master...
I20240503 14:11:55.661284 17077 heartbeater.cc:511] Master 127.16.94.189:39523 requested a full tablet report, sending...
I20240503 14:11:55.662598 16855 ts_manager.cc:194] Registered new tserver with Master: df5aa9fb874045759a08adf79fd4f0f7 (127.16.94.129:37917)
W20240503 14:11:55.664211 17085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:11:55.667330 16762 server_base.cc:1034] running on GCE node
I20240503 14:11:55.667629 16762 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:11:55.667704 16762 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:11:55.667796 16762 hybrid_clock.cc:648] HybridClock initialized: now 1714745515667753 us; error 0 us; skew 500 ppm
I20240503 14:11:55.667966 16762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:11:55.669387 16762 webserver.cc:460] Webserver started at http://127.16.94.130:35045/ using document root <none> and password file <none>
I20240503 14:11:55.669612 16762 fs_manager.cc:359] Metadata directory not provided
I20240503 14:11:55.669744 16762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:11:55.669878 16762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:11:55.670536 16762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714745515241994-16762-0/minicluster-data/ts-1-root/instance:
uuid: "eaa727f28b4f4380a69678d5bbb1eb3e"
format_stamp: "Formatted at 2024-05-03 14:11:55 on dist-test-slave-v27g"
I20240503 14:11:55.673214 16762 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20240503 14:11:55.675138 17090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:55.675511 16762 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 14:11:55.675621 16762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714745515241994-16762-0/minicluster-data/ts-1-root
uuid: "eaa727f28b4f4380a69678d5bbb1eb3e"
format_stamp: "Formatted at 2024-05-03 14:11:55 on dist-test-slave-v27g"
I20240503 14:11:55.675758 16762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714745515241994-16762-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714745515241994-16762-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714745515241994-16762-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:11:55.689100 16762 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:11:55.689782 16762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:11:55.690414 16762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:11:55.691372 16762 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:11:55.691447 16762 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:55.691568 16762 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:11:55.691627 16762 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:55.711270 16762 rpc_server.cc:307] RPC server started. Bound to: 127.16.94.130:43117
I20240503 14:11:55.711308 17153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.94.130:43117 every 8 connection(s)
I20240503 14:11:55.725749 16762 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 14:11:55.728590 16966 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc675fa1ed884b8bac510acadd18cd79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:11:55.728999 16966 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc675fa1ed884b8bac510acadd18cd79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9225ffbbab1a44839573abeb7b35f298" member_type: VOTER last_known_addr { host: "127.16.94.190" port: 41163 } } peers { permanent_uuid: "173b293a9078456ea4235d5503519f83" member_type: VOTER last_known_addr { host: "127.16.94.189" port: 39523 } } peers { permanent_uuid: "fc675fa1ed884b8bac510acadd18cd79" member_type: VOTER last_known_addr { host: "127.16.94.188" port: 42477 } }
I20240503 14:11:55.732983 17154 heartbeater.cc:347] Connected to a master server at 127.16.94.188:42477
I20240503 14:11:55.733151 17154 heartbeater.cc:464] Registering TS with master...
I20240503 14:11:55.733623 16966 leader_election.cc:288] T 00000000000000000000000000000000 P fc675fa1ed884b8bac510acadd18cd79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9225ffbbab1a44839573abeb7b35f298 (127.16.94.190:41163), 173b293a9078456ea4235d5503519f83 (127.16.94.189:39523)
I20240503 14:11:55.733906 17154 heartbeater.cc:511] Master 127.16.94.188:42477 requested a full tablet report, sending...
I20240503 14:11:55.734987 16802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fc675fa1ed884b8bac510acadd18cd79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9225ffbbab1a44839573abeb7b35f298" is_pre_election: true
I20240503 14:11:55.735502 16802 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9225ffbbab1a44839573abeb7b35f298 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc675fa1ed884b8bac510acadd18cd79 in term 0.
I20240503 14:11:55.736014 17155 heartbeater.cc:347] Connected to a master server at 127.16.94.190:41163
I20240503 14:11:55.736162 17155 heartbeater.cc:464] Registering TS with master...
I20240503 14:11:55.736472 16908 leader_election.cc:302] T 00000000000000000000000000000000 P fc675fa1ed884b8bac510acadd18cd79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9225ffbbab1a44839573abeb7b35f298, fc675fa1ed884b8bac510acadd18cd79; no voters:
I20240503 14:11:55.736778 17155 heartbeater.cc:511] Master 127.16.94.190:41163 requested a full tablet report, sending...
I20240503 14:11:55.738209 16923 ts_manager.cc:194] Registered new tserver with Master: eaa727f28b4f4380a69678d5bbb1eb3e (127.16.94.130:43117)
I20240503 14:11:55.738428 16868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fc675fa1ed884b8bac510acadd18cd79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "173b293a9078456ea4235d5503519f83" is_pre_election: true
I20240503 14:11:55.738754 16868 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 173b293a9078456ea4235d5503519f83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc675fa1ed884b8bac510acadd18cd79 in term 0.
I20240503 14:11:55.739002 17156 heartbeater.cc:347] Connected to a master server at 127.16.94.189:39523
I20240503 14:11:55.739159 17156 heartbeater.cc:464] Registering TS with master...
I20240503 14:11:55.739748 17156 heartbeater.cc:511] Master 127.16.94.189:39523 requested a full tablet report, sending...
I20240503 14:11:55.739842 16792 ts_manager.cc:194] Registered new tserver with Master: eaa727f28b4f4380a69678d5bbb1eb3e (127.16.94.130:43117)
I20240503 14:11:55.740624 16855 ts_manager.cc:194] Registered new tserver with Master: eaa727f28b4f4380a69678d5bbb1eb3e (127.16.94.130:43117)
I20240503 14:11:55.740945 16966 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc675fa1ed884b8bac510acadd18cd79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:11:55.741137 16966 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc675fa1ed884b8bac510acadd18cd79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:11:55.741266 16966 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc675fa1ed884b8bac510acadd18cd79 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:11:55.743721 16966 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc675fa1ed884b8bac510acadd18cd79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9225ffbbab1a44839573abeb7b35f298" member_type: VOTER last_known_addr { host: "127.16.94.190" port: 41163 } } peers { permanent_uuid: "173b293a9078456ea4235d5503519f83" member_type: VOTER last_known_addr { host: "127.16.94.189" port: 39523 } } peers { permanent_uuid: "fc675fa1ed884b8bac510acadd18cd79" member_type: VOTER last_known_addr { host: "127.16.94.188" port: 42477 } }
I20240503 14:11:55.744766 16868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fc675fa1ed884b8bac510acadd18cd79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "173b293a9078456ea4235d5503519f83"
I20240503 14:11:55.744962 16868 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 173b293a9078456ea4235d5503519f83 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:11:55.747154 16868 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 173b293a9078456ea4235d5503519f83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc675fa1ed884b8bac510acadd18cd79 in term 1.
I20240503 14:11:55.747522 16802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fc675fa1ed884b8bac510acadd18cd79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9225ffbbab1a44839573abeb7b35f298"
I20240503 14:11:55.747596 16907 leader_election.cc:302] T 00000000000000000000000000000000 P fc675fa1ed884b8bac510acadd18cd79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 173b293a9078456ea4235d5503519f83, fc675fa1ed884b8bac510acadd18cd79; no voters:
I20240503 14:11:55.747815 16802 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9225ffbbab1a44839573abeb7b35f298 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:11:55.751879 16802 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9225ffbbab1a44839573abeb7b35f298 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc675fa1ed884b8bac510acadd18cd79 in term 1.
I20240503 14:11:55.752650 16966 leader_election.cc:288] T 00000000000000000000000000000000 P fc675fa1ed884b8bac510acadd18cd79 [CANDIDATE]: Term 1 election: Requested vote from peers 9225ffbbab1a44839573abeb7b35f298 (127.16.94.190:41163), 173b293a9078456ea4235d5503519f83 (127.16.94.189:39523)
I20240503 14:11:55.752977 16966 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc675fa1ed884b8bac510acadd18cd79 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:11:55.753288 16966 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc675fa1ed884b8bac510acadd18cd79 [term 1 LEADER]: Becoming Leader. State: Replica: fc675fa1ed884b8bac510acadd18cd79, State: Running, Role: LEADER
I20240503 14:11:55.754247 16966 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc675fa1ed884b8bac510acadd18cd79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9225ffbbab1a44839573abeb7b35f298" member_type: VOTER last_known_addr { host: "127.16.94.190" port: 41163 } } peers { permanent_uuid: "173b293a9078456ea4235d5503519f83" member_type: VOTER last_known_addr { host: "127.16.94.189" port: 39523 } } peers { permanent_uuid: "fc675fa1ed884b8bac510acadd18cd79" member_type: VOTER last_known_addr { host: "127.16.94.188" port: 42477 } }
I20240503 14:11:55.760301 16966 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc675fa1ed884b8bac510acadd18cd79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc675fa1ed884b8bac510acadd18cd79. Latest consensus state: current_term: 1 leader_uuid: "fc675fa1ed884b8bac510acadd18cd79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9225ffbbab1a44839573abeb7b35f298" member_type: VOTER last_known_addr { host: "127.16.94.190" port: 41163 } } peers { permanent_uuid: "173b293a9078456ea4235d5503519f83" member_type: VOTER last_known_addr { host: "127.16.94.189" port: 39523 } } peers { permanent_uuid: "fc675fa1ed884b8bac510acadd18cd79" member_type: VOTER last_known_addr { host: "127.16.94.188" port: 42477 } } }
I20240503 14:11:55.760852 16966 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc675fa1ed884b8bac510acadd18cd79 [sys.catalog]: This master's current role is: LEADER
W20240503 14:11:55.762058 17161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:11:55.765548 17165 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 14:11:55.768764 17165 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20240503 14:11:55.774130 17163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:11:55.775290 16762 server_base.cc:1034] running on GCE node
W20240503 14:11:55.775707 17167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:11:55.776175 16762 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:11:55.776309 16762 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:11:55.776405 16762 hybrid_clock.cc:648] HybridClock initialized: now 1714745515776380 us; error 0 us; skew 500 ppm
I20240503 14:11:55.776638 16762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:11:55.779461 16802 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9225ffbbab1a44839573abeb7b35f298 [term 1 FOLLOWER]: Refusing update from remote peer fc675fa1ed884b8bac510acadd18cd79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:11:55.780457 16868 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 173b293a9078456ea4235d5503519f83 [term 1 FOLLOWER]: Refusing update from remote peer fc675fa1ed884b8bac510acadd18cd79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:11:55.780401 17162 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fc675fa1ed884b8bac510acadd18cd79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9225ffbbab1a44839573abeb7b35f298" member_type: VOTER last_known_addr { host: "127.16.94.190" port: 41163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:11:55.781267 16762 webserver.cc:460] Webserver started at http://127.16.94.131:43199/ using document root <none> and password file <none>
I20240503 14:11:55.781329 17162 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fc675fa1ed884b8bac510acadd18cd79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "173b293a9078456ea4235d5503519f83" member_type: VOTER last_known_addr { host: "127.16.94.189" port: 39523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:11:55.781548 16762 fs_manager.cc:359] Metadata directory not provided
I20240503 14:11:55.781684 16762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:11:55.781872 16762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:11:55.782727 16762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714745515241994-16762-0/minicluster-data/ts-2-root/instance:
uuid: "de2faf27a5c84a9588563a38a0e1eab2"
format_stamp: "Formatted at 2024-05-03 14:11:55 on dist-test-slave-v27g"
I20240503 14:11:55.795312 16967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 173b293a9078456ea4235d5503519f83 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc675fa1ed884b8bac510acadd18cd79. Latest consensus state: current_term: 1 leader_uuid: "fc675fa1ed884b8bac510acadd18cd79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9225ffbbab1a44839573abeb7b35f298" member_type: VOTER last_known_addr { host: "127.16.94.190" port: 41163 } } peers { permanent_uuid: "173b293a9078456ea4235d5503519f83" member_type: VOTER last_known_addr { host: "127.16.94.189" port: 39523 } } peers { permanent_uuid: "fc675fa1ed884b8bac510acadd18cd79" member_type: VOTER last_known_addr { host: "127.16.94.188" port: 42477 } } }
I20240503 14:11:55.795729 16967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 173b293a9078456ea4235d5503519f83 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 14:11:55.795563 16992 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9225ffbbab1a44839573abeb7b35f298 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc675fa1ed884b8bac510acadd18cd79. Latest consensus state: current_term: 1 leader_uuid: "fc675fa1ed884b8bac510acadd18cd79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9225ffbbab1a44839573abeb7b35f298" member_type: VOTER last_known_addr { host: "127.16.94.190" port: 41163 } } peers { permanent_uuid: "173b293a9078456ea4235d5503519f83" member_type: VOTER last_known_addr { host: "127.16.94.189" port: 39523 } } peers { permanent_uuid: "fc675fa1ed884b8bac510acadd18cd79" member_type: VOTER last_known_addr { host: "127.16.94.188" port: 42477 } } }
I20240503 14:11:55.795866 16992 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9225ffbbab1a44839573abeb7b35f298 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 14:11:55.798009 17169 mvcc.cc:205] Tried to move back new op lower bound from 7023597632620908544 to 7023597632543232000. Current Snapshot: MvccSnapshot[applied={T|T < 7023597632620908544}]
I20240503 14:11:55.800586 16762 fs_manager.cc:691] Time spent creating directory manager: real 0.017s user 0.009s sys 0.009s
I20240503 14:11:55.807533 16967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 173b293a9078456ea4235d5503519f83 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fc675fa1ed884b8bac510acadd18cd79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9225ffbbab1a44839573abeb7b35f298" member_type: VOTER last_known_addr { host: "127.16.94.190" port: 41163 } } peers { permanent_uuid: "173b293a9078456ea4235d5503519f83" member_type: VOTER last_known_addr { host: "127.16.94.189" port: 39523 } } peers { permanent_uuid: "fc675fa1ed884b8bac510acadd18cd79" member_type: VOTER last_known_addr { host: "127.16.94.188" port: 42477 } } }
I20240503 14:11:55.807850 16967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 173b293a9078456ea4235d5503519f83 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 14:11:55.809458 17180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:55.810019 16762 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.000s sys 0.001s
I20240503 14:11:55.810248 16762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714745515241994-16762-0/minicluster-data/ts-2-root
uuid: "de2faf27a5c84a9588563a38a0e1eab2"
format_stamp: "Formatted at 2024-05-03 14:11:55 on dist-test-slave-v27g"
I20240503 14:11:55.810497 16762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714745515241994-16762-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714745515241994-16762-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714745515241994-16762-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:11:55.811515 16966 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc675fa1ed884b8bac510acadd18cd79 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fc675fa1ed884b8bac510acadd18cd79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9225ffbbab1a44839573abeb7b35f298" member_type: VOTER last_known_addr { host: "127.16.94.190" port: 41163 } } peers { permanent_uuid: "173b293a9078456ea4235d5503519f83" member_type: VOTER last_known_addr { host: "127.16.94.189" port: 39523 } } peers { permanent_uuid: "fc675fa1ed884b8bac510acadd18cd79" member_type: VOTER last_known_addr { host: "127.16.94.188" port: 42477 } } }
I20240503 14:11:55.811765 16966 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc675fa1ed884b8bac510acadd18cd79 [sys.catalog]: This master's current role is: LEADER
I20240503 14:11:55.812013 16966 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc675fa1ed884b8bac510acadd18cd79 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fc675fa1ed884b8bac510acadd18cd79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9225ffbbab1a44839573abeb7b35f298" member_type: VOTER last_known_addr { host: "127.16.94.190" port: 41163 } } peers { permanent_uuid: "173b293a9078456ea4235d5503519f83" member_type: VOTER last_known_addr { host: "127.16.94.189" port: 39523 } } peers { permanent_uuid: "fc675fa1ed884b8bac510acadd18cd79" member_type: VOTER last_known_addr { host: "127.16.94.188" port: 42477 } } }
I20240503 14:11:55.812222 16966 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc675fa1ed884b8bac510acadd18cd79 [sys.catalog]: This master's current role is: LEADER
I20240503 14:11:55.813508 17165 catalog_manager.cc:1347] Generated new cluster ID: 2a2c221fc54e41a9b1b110c941b6d235
I20240503 14:11:55.813609 17165 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 14:11:55.826503 16992 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9225ffbbab1a44839573abeb7b35f298 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fc675fa1ed884b8bac510acadd18cd79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9225ffbbab1a44839573abeb7b35f298" member_type: VOTER last_known_addr { host: "127.16.94.190" port: 41163 } } peers { permanent_uuid: "173b293a9078456ea4235d5503519f83" member_type: VOTER last_known_addr { host: "127.16.94.189" port: 39523 } } peers { permanent_uuid: "fc675fa1ed884b8bac510acadd18cd79" member_type: VOTER last_known_addr { host: "127.16.94.188" port: 42477 } } }
I20240503 14:11:55.826773 16992 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9225ffbbab1a44839573abeb7b35f298 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 14:11:55.835575 17165 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 14:11:55.837306 17165 catalog_manager.cc:1504] Loading token signing keys...
I20240503 14:11:55.849107 16762 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:11:55.850194 16762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:11:55.853518 17165 catalog_manager.cc:5857] T 00000000000000000000000000000000 P fc675fa1ed884b8bac510acadd18cd79: Generated new TSK 0
I20240503 14:11:55.854148 16762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:11:55.854736 17165 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 14:11:55.855219 16762 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:11:55.855322 16762 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:55.855428 16762 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:11:55.855523 16762 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:55.883752 16762 rpc_server.cc:307] RPC server started. Bound to: 127.16.94.131:44293
I20240503 14:11:55.883782 17244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.94.131:44293 every 8 connection(s)
I20240503 14:11:55.918143 17245 heartbeater.cc:347] Connected to a master server at 127.16.94.188:42477
I20240503 14:11:55.918299 17247 heartbeater.cc:347] Connected to a master server at 127.16.94.189:39523
I20240503 14:11:55.918318 17245 heartbeater.cc:464] Registering TS with master...
I20240503 14:11:55.918500 17247 heartbeater.cc:464] Registering TS with master...
I20240503 14:11:55.919255 17245 heartbeater.cc:511] Master 127.16.94.188:42477 requested a full tablet report, sending...
I20240503 14:11:55.919910 17247 heartbeater.cc:511] Master 127.16.94.189:39523 requested a full tablet report, sending...
I20240503 14:11:55.920239 16923 ts_manager.cc:194] Registered new tserver with Master: de2faf27a5c84a9588563a38a0e1eab2 (127.16.94.131:44293)
I20240503 14:11:55.920521 17246 heartbeater.cc:347] Connected to a master server at 127.16.94.190:41163
I20240503 14:11:55.920670 17246 heartbeater.cc:464] Registering TS with master...
I20240503 14:11:55.920941 16855 ts_manager.cc:194] Registered new tserver with Master: de2faf27a5c84a9588563a38a0e1eab2 (127.16.94.131:44293)
I20240503 14:11:55.921240 17246 heartbeater.cc:511] Master 127.16.94.190:41163 requested a full tablet report, sending...
I20240503 14:11:55.922277 16792 ts_manager.cc:194] Registered new tserver with Master: de2faf27a5c84a9588563a38a0e1eab2 (127.16.94.131:44293)
I20240503 14:11:55.922567 16923 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48964
I20240503 14:11:55.922505 16762 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.010221652s
I20240503 14:11:56.560879 16987 catalog_manager.cc:1259] Loaded cluster ID: 2a2c221fc54e41a9b1b110c941b6d235
I20240503 14:11:56.560978 16987 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 173b293a9078456ea4235d5503519f83: loading cluster ID for follower catalog manager: success
I20240503 14:11:56.563078 16987 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 173b293a9078456ea4235d5503519f83: acquiring CA information for follower catalog manager: success
I20240503 14:11:56.564319 16987 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 173b293a9078456ea4235d5503519f83: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240503 14:11:56.569940 17003 catalog_manager.cc:1259] Loaded cluster ID: 2a2c221fc54e41a9b1b110c941b6d235
I20240503 14:11:56.570015 17003 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9225ffbbab1a44839573abeb7b35f298: loading cluster ID for follower catalog manager: success
I20240503 14:11:56.571766 17003 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9225ffbbab1a44839573abeb7b35f298: acquiring CA information for follower catalog manager: success
I20240503 14:11:56.572762 17003 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9225ffbbab1a44839573abeb7b35f298: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240503 14:11:56.660967 16923 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48952
I20240503 14:11:56.743290 16923 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48956
I20240503 14:11:56.924607 17245 heartbeater.cc:503] Master 127.16.94.188:42477 was elected leader, sending a full tablet report...
I20240503 14:11:57.553254 16988 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:11:57.662197 17075 heartbeater.cc:503] Master 127.16.94.188:42477 was elected leader, sending a full tablet report...
I20240503 14:11:57.744347 17154 heartbeater.cc:503] Master 127.16.94.188:42477 was elected leader, sending a full tablet report...
I20240503 14:11:57.985947 16762 test_util.cc:268] Using random seed: 1299894398
I20240503 14:11:58.023655 16923 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48976:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:11:58.058514 17209 tablet_service.cc:1467] Processing CreateTablet for tablet 11cb8ca56e2547fe952aad52dd2e9cb8 (DEFAULT_TABLE table=test-workload [id=28841e23edb84735b819cc0a569fca23]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240503 14:11:58.059489 17209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11cb8ca56e2547fe952aad52dd2e9cb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:11:58.061712 17040 tablet_service.cc:1467] Processing CreateTablet for tablet e012b7d7628c4ab8bed2a9aae0bac849 (DEFAULT_TABLE table=test-workload [id=28841e23edb84735b819cc0a569fca23]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240503 14:11:58.061914 17039 tablet_service.cc:1467] Processing CreateTablet for tablet 7521adb4793747ce8e9035c4491c2060 (DEFAULT_TABLE table=test-workload [id=28841e23edb84735b819cc0a569fca23]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240503 14:11:58.062193 17040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e012b7d7628c4ab8bed2a9aae0bac849. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:11:58.062512 17039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7521adb4793747ce8e9035c4491c2060. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:11:58.063453 17119 tablet_service.cc:1467] Processing CreateTablet for tablet e0f5dfe28a9d4419a5f376f37210179a (DEFAULT_TABLE table=test-workload [id=28841e23edb84735b819cc0a569fca23]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240503 14:11:58.064070 17119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0f5dfe28a9d4419a5f376f37210179a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:11:58.072223 17282 tablet_bootstrap.cc:492] T e0f5dfe28a9d4419a5f376f37210179a P eaa727f28b4f4380a69678d5bbb1eb3e: Bootstrap starting.
I20240503 14:11:58.073064 17280 tablet_bootstrap.cc:492] T 11cb8ca56e2547fe952aad52dd2e9cb8 P de2faf27a5c84a9588563a38a0e1eab2: Bootstrap starting.
I20240503 14:11:58.074298 17282 tablet_bootstrap.cc:654] T e0f5dfe28a9d4419a5f376f37210179a P eaa727f28b4f4380a69678d5bbb1eb3e: Neither blocks nor log segments found. Creating new log.
I20240503 14:11:58.076157 17280 tablet_bootstrap.cc:654] T 11cb8ca56e2547fe952aad52dd2e9cb8 P de2faf27a5c84a9588563a38a0e1eab2: Neither blocks nor log segments found. Creating new log.
I20240503 14:11:58.076691 17282 tablet_bootstrap.cc:492] T e0f5dfe28a9d4419a5f376f37210179a P eaa727f28b4f4380a69678d5bbb1eb3e: No bootstrap required, opened a new log
I20240503 14:11:58.076867 17282 ts_tablet_manager.cc:1367] T e0f5dfe28a9d4419a5f376f37210179a P eaa727f28b4f4380a69678d5bbb1eb3e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 14:11:58.078022 17282 raft_consensus.cc:348] T e0f5dfe28a9d4419a5f376f37210179a P eaa727f28b4f4380a69678d5bbb1eb3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaa727f28b4f4380a69678d5bbb1eb3e" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 43117 } }
I20240503 14:11:58.078267 17282 raft_consensus.cc:374] T e0f5dfe28a9d4419a5f376f37210179a P eaa727f28b4f4380a69678d5bbb1eb3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:11:58.078466 17282 raft_consensus.cc:729] T e0f5dfe28a9d4419a5f376f37210179a P eaa727f28b4f4380a69678d5bbb1eb3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaa727f28b4f4380a69678d5bbb1eb3e, State: Initialized, Role: FOLLOWER
I20240503 14:11:58.078697 17281 tablet_bootstrap.cc:492] T e012b7d7628c4ab8bed2a9aae0bac849 P df5aa9fb874045759a08adf79fd4f0f7: Bootstrap starting.
I20240503 14:11:58.078797 17282 consensus_queue.cc:260] T e0f5dfe28a9d4419a5f376f37210179a P eaa727f28b4f4380a69678d5bbb1eb3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaa727f28b4f4380a69678d5bbb1eb3e" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 43117 } }
I20240503 14:11:58.078939 17280 tablet_bootstrap.cc:492] T 11cb8ca56e2547fe952aad52dd2e9cb8 P de2faf27a5c84a9588563a38a0e1eab2: No bootstrap required, opened a new log
I20240503 14:11:58.079059 17282 raft_consensus.cc:388] T e0f5dfe28a9d4419a5f376f37210179a P eaa727f28b4f4380a69678d5bbb1eb3e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:11:58.079100 17280 ts_tablet_manager.cc:1367] T 11cb8ca56e2547fe952aad52dd2e9cb8 P de2faf27a5c84a9588563a38a0e1eab2: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240503 14:11:58.079175 17282 raft_consensus.cc:482] T e0f5dfe28a9d4419a5f376f37210179a P eaa727f28b4f4380a69678d5bbb1eb3e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:11:58.079303 17282 raft_consensus.cc:3037] T e0f5dfe28a9d4419a5f376f37210179a P eaa727f28b4f4380a69678d5bbb1eb3e [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:11:58.079946 17280 raft_consensus.cc:348] T 11cb8ca56e2547fe952aad52dd2e9cb8 P de2faf27a5c84a9588563a38a0e1eab2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de2faf27a5c84a9588563a38a0e1eab2" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44293 } }
I20240503 14:11:58.080152 17280 raft_consensus.cc:374] T 11cb8ca56e2547fe952aad52dd2e9cb8 P de2faf27a5c84a9588563a38a0e1eab2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:11:58.080269 17280 raft_consensus.cc:729] T 11cb8ca56e2547fe952aad52dd2e9cb8 P de2faf27a5c84a9588563a38a0e1eab2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de2faf27a5c84a9588563a38a0e1eab2, State: Initialized, Role: FOLLOWER
I20240503 14:11:58.080498 17280 consensus_queue.cc:260] T 11cb8ca56e2547fe952aad52dd2e9cb8 P de2faf27a5c84a9588563a38a0e1eab2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de2faf27a5c84a9588563a38a0e1eab2" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44293 } }
I20240503 14:11:58.080726 17281 tablet_bootstrap.cc:654] T e012b7d7628c4ab8bed2a9aae0bac849 P df5aa9fb874045759a08adf79fd4f0f7: Neither blocks nor log segments found. Creating new log.
I20240503 14:11:58.080732 17280 raft_consensus.cc:388] T 11cb8ca56e2547fe952aad52dd2e9cb8 P de2faf27a5c84a9588563a38a0e1eab2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:11:58.080849 17280 raft_consensus.cc:482] T 11cb8ca56e2547fe952aad52dd2e9cb8 P de2faf27a5c84a9588563a38a0e1eab2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:11:58.080993 17280 raft_consensus.cc:3037] T 11cb8ca56e2547fe952aad52dd2e9cb8 P de2faf27a5c84a9588563a38a0e1eab2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:11:58.082166 17282 raft_consensus.cc:504] T e0f5dfe28a9d4419a5f376f37210179a P eaa727f28b4f4380a69678d5bbb1eb3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaa727f28b4f4380a69678d5bbb1eb3e" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 43117 } }
I20240503 14:11:58.082494 17282 leader_election.cc:302] T e0f5dfe28a9d4419a5f376f37210179a P eaa727f28b4f4380a69678d5bbb1eb3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eaa727f28b4f4380a69678d5bbb1eb3e; no voters:
I20240503 14:11:58.083359 17281 tablet_bootstrap.cc:492] T e012b7d7628c4ab8bed2a9aae0bac849 P df5aa9fb874045759a08adf79fd4f0f7: No bootstrap required, opened a new log
I20240503 14:11:58.083494 17281 ts_tablet_manager.cc:1367] T e012b7d7628c4ab8bed2a9aae0bac849 P df5aa9fb874045759a08adf79fd4f0f7: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20240503 14:11:58.083750 17280 raft_consensus.cc:504] T 11cb8ca56e2547fe952aad52dd2e9cb8 P de2faf27a5c84a9588563a38a0e1eab2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de2faf27a5c84a9588563a38a0e1eab2" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44293 } }
I20240503 14:11:58.083981 17280 leader_election.cc:302] T 11cb8ca56e2547fe952aad52dd2e9cb8 P de2faf27a5c84a9588563a38a0e1eab2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de2faf27a5c84a9588563a38a0e1eab2; no voters:
I20240503 14:11:58.084287 17281 raft_consensus.cc:348] T e012b7d7628c4ab8bed2a9aae0bac849 P df5aa9fb874045759a08adf79fd4f0f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df5aa9fb874045759a08adf79fd4f0f7" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37917 } }
I20240503 14:11:58.084630 17281 raft_consensus.cc:374] T e012b7d7628c4ab8bed2a9aae0bac849 P df5aa9fb874045759a08adf79fd4f0f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:11:58.084709 17280 leader_election.cc:288] T 11cb8ca56e2547fe952aad52dd2e9cb8 P de2faf27a5c84a9588563a38a0e1eab2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:11:58.084749 17281 raft_consensus.cc:729] T e012b7d7628c4ab8bed2a9aae0bac849 P df5aa9fb874045759a08adf79fd4f0f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df5aa9fb874045759a08adf79fd4f0f7, State: Initialized, Role: FOLLOWER
I20240503 14:11:58.085022 17281 consensus_queue.cc:260] T e012b7d7628c4ab8bed2a9aae0bac849 P df5aa9fb874045759a08adf79fd4f0f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df5aa9fb874045759a08adf79fd4f0f7" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37917 } }
I20240503 14:11:58.085224 17287 raft_consensus.cc:2781] T 11cb8ca56e2547fe952aad52dd2e9cb8 P de2faf27a5c84a9588563a38a0e1eab2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:11:58.085358 17281 raft_consensus.cc:388] T e012b7d7628c4ab8bed2a9aae0bac849 P df5aa9fb874045759a08adf79fd4f0f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:11:58.085476 17281 raft_consensus.cc:482] T e012b7d7628c4ab8bed2a9aae0bac849 P df5aa9fb874045759a08adf79fd4f0f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:11:58.085644 17281 raft_consensus.cc:3037] T e012b7d7628c4ab8bed2a9aae0bac849 P df5aa9fb874045759a08adf79fd4f0f7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:11:58.085969 17280 ts_tablet_manager.cc:1398] T 11cb8ca56e2547fe952aad52dd2e9cb8 P de2faf27a5c84a9588563a38a0e1eab2: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240503 14:11:58.090055 17281 raft_consensus.cc:504] T e012b7d7628c4ab8bed2a9aae0bac849 P df5aa9fb874045759a08adf79fd4f0f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df5aa9fb874045759a08adf79fd4f0f7" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37917 } }
I20240503 14:11:58.090149 17282 leader_election.cc:288] T e0f5dfe28a9d4419a5f376f37210179a P eaa727f28b4f4380a69678d5bbb1eb3e [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:11:58.090303 17286 raft_consensus.cc:2781] T e0f5dfe28a9d4419a5f376f37210179a P eaa727f28b4f4380a69678d5bbb1eb3e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:11:58.090364 17281 leader_election.cc:302] T e012b7d7628c4ab8bed2a9aae0bac849 P df5aa9fb874045759a08adf79fd4f0f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df5aa9fb874045759a08adf79fd4f0f7; no voters:
I20240503 14:11:58.090700 17287 raft_consensus.cc:686] T 11cb8ca56e2547fe952aad52dd2e9cb8 P de2faf27a5c84a9588563a38a0e1eab2 [term 1 LEADER]: Becoming Leader. State: Replica: de2faf27a5c84a9588563a38a0e1eab2, State: Running, Role: LEADER
I20240503 14:11:58.091167 17281 leader_election.cc:288] T e012b7d7628c4ab8bed2a9aae0bac849 P df5aa9fb874045759a08adf79fd4f0f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:11:58.091172 17287 consensus_queue.cc:237] T 11cb8ca56e2547fe952aad52dd2e9cb8 P de2faf27a5c84a9588563a38a0e1eab2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de2faf27a5c84a9588563a38a0e1eab2" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44293 } }
I20240503 14:11:58.092217 17291 raft_consensus.cc:2781] T e012b7d7628c4ab8bed2a9aae0bac849 P df5aa9fb874045759a08adf79fd4f0f7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:11:58.092998 17281 ts_tablet_manager.cc:1398] T e012b7d7628c4ab8bed2a9aae0bac849 P df5aa9fb874045759a08adf79fd4f0f7: Time spent starting tablet: real 0.009s user 0.007s sys 0.001s
I20240503 14:11:58.093386 17286 raft_consensus.cc:686] T e0f5dfe28a9d4419a5f376f37210179a P eaa727f28b4f4380a69678d5bbb1eb3e [term 1 LEADER]: Becoming Leader. State: Replica: eaa727f28b4f4380a69678d5bbb1eb3e, State: Running, Role: LEADER
I20240503 14:11:58.093492 17281 tablet_bootstrap.cc:492] T 7521adb4793747ce8e9035c4491c2060 P df5aa9fb874045759a08adf79fd4f0f7: Bootstrap starting.
I20240503 14:11:58.093693 17282 ts_tablet_manager.cc:1398] T e0f5dfe28a9d4419a5f376f37210179a P eaa727f28b4f4380a69678d5bbb1eb3e: Time spent starting tablet: real 0.017s user 0.007s sys 0.008s
I20240503 14:11:58.094337 17286 consensus_queue.cc:237] T e0f5dfe28a9d4419a5f376f37210179a P eaa727f28b4f4380a69678d5bbb1eb3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaa727f28b4f4380a69678d5bbb1eb3e" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 43117 } }
I20240503 14:11:58.101140 17281 tablet_bootstrap.cc:654] T 7521adb4793747ce8e9035c4491c2060 P df5aa9fb874045759a08adf79fd4f0f7: Neither blocks nor log segments found. Creating new log.
I20240503 14:11:58.104924 17291 raft_consensus.cc:686] T e012b7d7628c4ab8bed2a9aae0bac849 P df5aa9fb874045759a08adf79fd4f0f7 [term 1 LEADER]: Becoming Leader. State: Replica: df5aa9fb874045759a08adf79fd4f0f7, State: Running, Role: LEADER
I20240503 14:11:58.105365 17291 consensus_queue.cc:237] T e012b7d7628c4ab8bed2a9aae0bac849 P df5aa9fb874045759a08adf79fd4f0f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df5aa9fb874045759a08adf79fd4f0f7" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37917 } }
I20240503 14:11:58.110004 17281 tablet_bootstrap.cc:492] T 7521adb4793747ce8e9035c4491c2060 P df5aa9fb874045759a08adf79fd4f0f7: No bootstrap required, opened a new log
I20240503 14:11:58.111173 17281 ts_tablet_manager.cc:1367] T 7521adb4793747ce8e9035c4491c2060 P df5aa9fb874045759a08adf79fd4f0f7: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.003s
I20240503 14:11:58.112321 17281 raft_consensus.cc:348] T 7521adb4793747ce8e9035c4491c2060 P df5aa9fb874045759a08adf79fd4f0f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df5aa9fb874045759a08adf79fd4f0f7" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37917 } }
I20240503 14:11:58.112565 17281 raft_consensus.cc:374] T 7521adb4793747ce8e9035c4491c2060 P df5aa9fb874045759a08adf79fd4f0f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:11:58.112671 17281 raft_consensus.cc:729] T 7521adb4793747ce8e9035c4491c2060 P df5aa9fb874045759a08adf79fd4f0f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df5aa9fb874045759a08adf79fd4f0f7, State: Initialized, Role: FOLLOWER
I20240503 14:11:58.112969 17281 consensus_queue.cc:260] T 7521adb4793747ce8e9035c4491c2060 P df5aa9fb874045759a08adf79fd4f0f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df5aa9fb874045759a08adf79fd4f0f7" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37917 } }
I20240503 14:11:58.113204 17281 raft_consensus.cc:388] T 7521adb4793747ce8e9035c4491c2060 P df5aa9fb874045759a08adf79fd4f0f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:11:58.113322 17281 raft_consensus.cc:482] T 7521adb4793747ce8e9035c4491c2060 P df5aa9fb874045759a08adf79fd4f0f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:11:58.113435 17281 raft_consensus.cc:3037] T 7521adb4793747ce8e9035c4491c2060 P df5aa9fb874045759a08adf79fd4f0f7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:11:58.116771 16923 catalog_manager.cc:5484] T 11cb8ca56e2547fe952aad52dd2e9cb8 P de2faf27a5c84a9588563a38a0e1eab2 reported cstate change: term changed from 0 to 1, leader changed from <none> to de2faf27a5c84a9588563a38a0e1eab2 (127.16.94.131). New cstate: current_term: 1 leader_uuid: "de2faf27a5c84a9588563a38a0e1eab2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de2faf27a5c84a9588563a38a0e1eab2" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44293 } health_report { overall_health: HEALTHY } } }
I20240503 14:11:58.118551 16922 catalog_manager.cc:5484] T e0f5dfe28a9d4419a5f376f37210179a P eaa727f28b4f4380a69678d5bbb1eb3e reported cstate change: term changed from 0 to 1, leader changed from <none> to eaa727f28b4f4380a69678d5bbb1eb3e (127.16.94.130). New cstate: current_term: 1 leader_uuid: "eaa727f28b4f4380a69678d5bbb1eb3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaa727f28b4f4380a69678d5bbb1eb3e" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 43117 } health_report { overall_health: HEALTHY } } }
I20240503 14:11:58.130020 17281 raft_consensus.cc:504] T 7521adb4793747ce8e9035c4491c2060 P df5aa9fb874045759a08adf79fd4f0f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df5aa9fb874045759a08adf79fd4f0f7" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37917 } }
I20240503 14:11:58.130355 17281 leader_election.cc:302] T 7521adb4793747ce8e9035c4491c2060 P df5aa9fb874045759a08adf79fd4f0f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df5aa9fb874045759a08adf79fd4f0f7; no voters:
I20240503 14:11:58.130631 17281 leader_election.cc:288] T 7521adb4793747ce8e9035c4491c2060 P df5aa9fb874045759a08adf79fd4f0f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:11:58.130903 17294 raft_consensus.cc:2781] T 7521adb4793747ce8e9035c4491c2060 P df5aa9fb874045759a08adf79fd4f0f7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:11:58.137254 17281 ts_tablet_manager.cc:1398] T 7521adb4793747ce8e9035c4491c2060 P df5aa9fb874045759a08adf79fd4f0f7: Time spent starting tablet: real 0.026s user 0.002s sys 0.005s
I20240503 14:11:58.137940 17294 raft_consensus.cc:686] T 7521adb4793747ce8e9035c4491c2060 P df5aa9fb874045759a08adf79fd4f0f7 [term 1 LEADER]: Becoming Leader. State: Replica: df5aa9fb874045759a08adf79fd4f0f7, State: Running, Role: LEADER
I20240503 14:11:58.138321 17294 consensus_queue.cc:237] T 7521adb4793747ce8e9035c4491c2060 P df5aa9fb874045759a08adf79fd4f0f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df5aa9fb874045759a08adf79fd4f0f7" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37917 } }
I20240503 14:11:58.140087 16921 catalog_manager.cc:5484] T e012b7d7628c4ab8bed2a9aae0bac849 P df5aa9fb874045759a08adf79fd4f0f7 reported cstate change: term changed from 0 to 1, leader changed from <none> to df5aa9fb874045759a08adf79fd4f0f7 (127.16.94.129). New cstate: current_term: 1 leader_uuid: "df5aa9fb874045759a08adf79fd4f0f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df5aa9fb874045759a08adf79fd4f0f7" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37917 } health_report { overall_health: HEALTHY } } }
I20240503 14:11:58.151454 16923 catalog_manager.cc:5484] T 7521adb4793747ce8e9035c4491c2060 P df5aa9fb874045759a08adf79fd4f0f7 reported cstate change: term changed from 0 to 1, leader changed from <none> to df5aa9fb874045759a08adf79fd4f0f7 (127.16.94.129). New cstate: current_term: 1 leader_uuid: "df5aa9fb874045759a08adf79fd4f0f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df5aa9fb874045759a08adf79fd4f0f7" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37917 } health_report { overall_health: HEALTHY } } }
I20240503 14:11:58.167147 16762 tablet_server.cc:178] TabletServer@127.16.94.129:0 shutting down...
I20240503 14:11:58.178341 16762 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 14:11:58.178745 16762 tablet_replica.cc:330] T 7521adb4793747ce8e9035c4491c2060 P df5aa9fb874045759a08adf79fd4f0f7: stopping tablet replica
I20240503 14:11:58.179121 16762 raft_consensus.cc:2229] T 7521adb4793747ce8e9035c4491c2060 P df5aa9fb874045759a08adf79fd4f0f7 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:11:58.179388 16762 raft_consensus.cc:2258] T 7521adb4793747ce8e9035c4491c2060 P df5aa9fb874045759a08adf79fd4f0f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:11:58.180838 16762 tablet_replica.cc:330] T e012b7d7628c4ab8bed2a9aae0bac849 P df5aa9fb874045759a08adf79fd4f0f7: stopping tablet replica
I20240503 14:11:58.180995 16762 raft_consensus.cc:2229] T e012b7d7628c4ab8bed2a9aae0bac849 P df5aa9fb874045759a08adf79fd4f0f7 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:11:58.181172 16762 raft_consensus.cc:2258] T e012b7d7628c4ab8bed2a9aae0bac849 P df5aa9fb874045759a08adf79fd4f0f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:11:58.196413 16762 tablet_server.cc:194] TabletServer@127.16.94.129:0 shutdown complete.
I20240503 14:11:58.201934 16762 tablet_server.cc:178] TabletServer@127.16.94.130:0 shutting down...
I20240503 14:11:58.210659 16762 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 14:11:58.211122 16762 tablet_replica.cc:330] T e0f5dfe28a9d4419a5f376f37210179a P eaa727f28b4f4380a69678d5bbb1eb3e: stopping tablet replica
I20240503 14:11:58.211301 16762 raft_consensus.cc:2229] T e0f5dfe28a9d4419a5f376f37210179a P eaa727f28b4f4380a69678d5bbb1eb3e [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:11:58.211462 16762 raft_consensus.cc:2258] T e0f5dfe28a9d4419a5f376f37210179a P eaa727f28b4f4380a69678d5bbb1eb3e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:11:58.225534 16762 tablet_server.cc:194] TabletServer@127.16.94.130:0 shutdown complete.
I20240503 14:11:58.228286 16762 tablet_server.cc:178] TabletServer@127.16.94.131:0 shutting down...
I20240503 14:11:58.236387 16762 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 14:11:58.236755 16762 tablet_replica.cc:330] T 11cb8ca56e2547fe952aad52dd2e9cb8 P de2faf27a5c84a9588563a38a0e1eab2: stopping tablet replica
I20240503 14:11:58.236974 16762 raft_consensus.cc:2229] T 11cb8ca56e2547fe952aad52dd2e9cb8 P de2faf27a5c84a9588563a38a0e1eab2 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:11:58.237175 16762 raft_consensus.cc:2258] T 11cb8ca56e2547fe952aad52dd2e9cb8 P de2faf27a5c84a9588563a38a0e1eab2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:11:58.251185 16762 tablet_server.cc:194] TabletServer@127.16.94.131:0 shutdown complete.
I20240503 14:11:58.254240 16762 master.cc:537] Master@127.16.94.190:41163 shutting down...
I20240503 14:11:58.262647 16762 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 9225ffbbab1a44839573abeb7b35f298 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:11:58.262892 16762 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 9225ffbbab1a44839573abeb7b35f298 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:11:58.263048 16762 tablet_replica.cc:330] T 00000000000000000000000000000000 P 9225ffbbab1a44839573abeb7b35f298: stopping tablet replica
I20240503 14:11:58.269984 16762 master.cc:552] Master@127.16.94.190:41163 shutdown complete.
I20240503 14:11:58.275152 16762 master.cc:537] Master@127.16.94.189:39523 shutting down...
I20240503 14:11:58.282320 16762 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 173b293a9078456ea4235d5503519f83 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:11:58.282567 16762 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 173b293a9078456ea4235d5503519f83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:11:58.282697 16762 tablet_replica.cc:330] T 00000000000000000000000000000000 P 173b293a9078456ea4235d5503519f83: stopping tablet replica
I20240503 14:11:58.296540 16762 master.cc:552] Master@127.16.94.189:39523 shutdown complete.
I20240503 14:11:58.300398 16762 master.cc:537] Master@127.16.94.188:42477 shutting down...
I20240503 14:11:58.307821 16762 raft_consensus.cc:2229] T 00000000000000000000000000000000 P fc675fa1ed884b8bac510acadd18cd79 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:11:58.308259 16762 raft_consensus.cc:2258] T 00000000000000000000000000000000 P fc675fa1ed884b8bac510acadd18cd79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:11:58.308374 16762 tablet_replica.cc:330] T 00000000000000000000000000000000 P fc675fa1ed884b8bac510acadd18cd79: stopping tablet replica
I20240503 14:11:58.322578 16762 master.cc:552] Master@127.16.94.188:42477 shutdown complete.
[ OK ] AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing (3025 ms)
[ RUN ] AutoRebalancerTest.AutoRebalancingTurnOffAndOn
I20240503 14:11:58.336050 16762 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.16.94.190:44937
I20240503 14:11:58.336505 16762 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:11:58.339284 17304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:11:58.339886 17305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:11:58.340560 17307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:11:58.341209 16762 server_base.cc:1034] running on GCE node
I20240503 14:11:58.341457 16762 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:11:58.341538 16762 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:11:58.341596 16762 hybrid_clock.cc:648] HybridClock initialized: now 1714745518341578 us; error 0 us; skew 500 ppm
I20240503 14:11:58.341796 16762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:11:58.343005 16762 webserver.cc:460] Webserver started at http://127.16.94.190:46297/ using document root <none> and password file <none>
I20240503 14:11:58.343235 16762 fs_manager.cc:359] Metadata directory not provided
I20240503 14:11:58.343325 16762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:11:58.343473 16762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:11:58.344096 16762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714745515241994-16762-0/minicluster-data/master-0-root/instance:
uuid: "ed158d547a714aa1be8b2e669157b58e"
format_stamp: "Formatted at 2024-05-03 14:11:58 on dist-test-slave-v27g"
I20240503 14:11:58.346603 16762 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.002s
I20240503 14:11:58.348109 17312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:58.348440 16762 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240503 14:11:58.348544 16762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714745515241994-16762-0/minicluster-data/master-0-root
uuid: "ed158d547a714aa1be8b2e669157b58e"
format_stamp: "Formatted at 2024-05-03 14:11:58 on dist-test-slave-v27g"
I20240503 14:11:58.348657 16762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714745515241994-16762-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714745515241994-16762-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714745515241994-16762-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:11:58.362159 16762 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:11:58.362717 16762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:11:58.376638 16762 rpc_server.cc:307] RPC server started. Bound to: 127.16.94.190:44937
I20240503 14:11:58.376660 17363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.94.190:44937 every 8 connection(s)
I20240503 14:11:58.377986 17364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:11:58.381881 17364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed158d547a714aa1be8b2e669157b58e: Bootstrap starting.
I20240503 14:11:58.383102 17364 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed158d547a714aa1be8b2e669157b58e: Neither blocks nor log segments found. Creating new log.
I20240503 14:11:58.384670 17364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed158d547a714aa1be8b2e669157b58e: No bootstrap required, opened a new log
I20240503 14:11:58.385269 17364 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed158d547a714aa1be8b2e669157b58e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed158d547a714aa1be8b2e669157b58e" member_type: VOTER }
I20240503 14:11:58.385401 17364 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed158d547a714aa1be8b2e669157b58e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:11:58.385494 17364 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed158d547a714aa1be8b2e669157b58e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed158d547a714aa1be8b2e669157b58e, State: Initialized, Role: FOLLOWER
I20240503 14:11:58.385747 17364 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed158d547a714aa1be8b2e669157b58e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed158d547a714aa1be8b2e669157b58e" member_type: VOTER }
I20240503 14:11:58.385893 17364 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed158d547a714aa1be8b2e669157b58e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:11:58.385968 17364 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed158d547a714aa1be8b2e669157b58e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:11:58.386081 17364 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed158d547a714aa1be8b2e669157b58e [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:11:58.387830 17364 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed158d547a714aa1be8b2e669157b58e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed158d547a714aa1be8b2e669157b58e" member_type: VOTER }
I20240503 14:11:58.388052 17364 leader_election.cc:302] T 00000000000000000000000000000000 P ed158d547a714aa1be8b2e669157b58e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed158d547a714aa1be8b2e669157b58e; no voters:
I20240503 14:11:58.388409 17364 leader_election.cc:288] T 00000000000000000000000000000000 P ed158d547a714aa1be8b2e669157b58e [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:11:58.388564 17367 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed158d547a714aa1be8b2e669157b58e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:11:58.389060 17367 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed158d547a714aa1be8b2e669157b58e [term 1 LEADER]: Becoming Leader. State: Replica: ed158d547a714aa1be8b2e669157b58e, State: Running, Role: LEADER
I20240503 14:11:58.389310 17364 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed158d547a714aa1be8b2e669157b58e [sys.catalog]: configured and running, proceeding with master startup.
I20240503 14:11:58.389350 17367 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed158d547a714aa1be8b2e669157b58e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed158d547a714aa1be8b2e669157b58e" member_type: VOTER }
I20240503 14:11:58.390362 17368 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed158d547a714aa1be8b2e669157b58e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed158d547a714aa1be8b2e669157b58e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed158d547a714aa1be8b2e669157b58e" member_type: VOTER } }
I20240503 14:11:58.390437 17369 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed158d547a714aa1be8b2e669157b58e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed158d547a714aa1be8b2e669157b58e. Latest consensus state: current_term: 1 leader_uuid: "ed158d547a714aa1be8b2e669157b58e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed158d547a714aa1be8b2e669157b58e" member_type: VOTER } }
I20240503 14:11:58.390762 17369 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed158d547a714aa1be8b2e669157b58e [sys.catalog]: This master's current role is: LEADER
I20240503 14:11:58.390780 17368 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed158d547a714aa1be8b2e669157b58e [sys.catalog]: This master's current role is: LEADER
I20240503 14:11:58.391903 17372 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 14:11:58.393371 17372 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 14:11:58.394814 16762 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 14:11:58.396270 17372 catalog_manager.cc:1347] Generated new cluster ID: 6d2722990a394c1fa33fbc6efbe635fe
I20240503 14:11:58.396344 17372 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 14:11:58.403005 17372 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 14:11:58.403693 17372 catalog_manager.cc:1504] Loading token signing keys...
I20240503 14:11:58.415925 17372 catalog_manager.cc:5857] T 00000000000000000000000000000000 P ed158d547a714aa1be8b2e669157b58e: Generated new TSK 0
I20240503 14:11:58.416205 17372 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 14:11:58.427544 16762 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:11:58.430825 17385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:11:58.431470 17386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:11:58.432204 17388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:11:58.432868 16762 server_base.cc:1034] running on GCE node
I20240503 14:11:58.433146 16762 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:11:58.433209 16762 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:11:58.433280 16762 hybrid_clock.cc:648] HybridClock initialized: now 1714745518433247 us; error 0 us; skew 500 ppm
I20240503 14:11:58.433430 16762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:11:58.434646 16762 webserver.cc:460] Webserver started at http://127.16.94.129:38685/ using document root <none> and password file <none>
I20240503 14:11:58.434867 16762 fs_manager.cc:359] Metadata directory not provided
I20240503 14:11:58.434958 16762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:11:58.435098 16762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:11:58.435736 16762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714745515241994-16762-0/minicluster-data/ts-0-root/instance:
uuid: "6ad404efe189443cb1058eaf35888347"
format_stamp: "Formatted at 2024-05-03 14:11:58 on dist-test-slave-v27g"
I20240503 14:11:58.438206 16762 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20240503 14:11:58.439729 17393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:58.440162 16762 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240503 14:11:58.440281 16762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714745515241994-16762-0/minicluster-data/ts-0-root
uuid: "6ad404efe189443cb1058eaf35888347"
format_stamp: "Formatted at 2024-05-03 14:11:58 on dist-test-slave-v27g"
I20240503 14:11:58.440426 16762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714745515241994-16762-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714745515241994-16762-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714745515241994-16762-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:11:58.446095 16762 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:11:58.446615 16762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:11:58.447144 16762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:11:58.447971 16762 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:11:58.448040 16762 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:58.448146 16762 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:11:58.448196 16762 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:58.463889 16762 rpc_server.cc:307] RPC server started. Bound to: 127.16.94.129:42357
I20240503 14:11:58.463945 17456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.94.129:42357 every 8 connection(s)
I20240503 14:11:58.465683 16762 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:11:58.469725 17461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:11:58.470271 17462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:11:58.470937 17464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:11:58.473793 16762 server_base.cc:1034] running on GCE node
I20240503 14:11:58.474144 16762 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:11:58.474270 16762 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:11:58.474366 16762 hybrid_clock.cc:648] HybridClock initialized: now 1714745518474327 us; error 0 us; skew 500 ppm
I20240503 14:11:58.474573 16762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:11:58.475478 17457 heartbeater.cc:347] Connected to a master server at 127.16.94.190:44937
I20240503 14:11:58.475636 17457 heartbeater.cc:464] Registering TS with master...
I20240503 14:11:58.476148 17457 heartbeater.cc:511] Master 127.16.94.190:44937 requested a full tablet report, sending...
I20240503 14:11:58.476244 16762 webserver.cc:460] Webserver started at http://127.16.94.130:41113/ using document root <none> and password file <none>
I20240503 14:11:58.476529 16762 fs_manager.cc:359] Metadata directory not provided
I20240503 14:11:58.476663 16762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:11:58.476966 16762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:11:58.477097 17329 ts_manager.cc:194] Registered new tserver with Master: 6ad404efe189443cb1058eaf35888347 (127.16.94.129:42357)
I20240503 14:11:58.477632 16762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714745515241994-16762-0/minicluster-data/ts-1-root/instance:
uuid: "09f6d631be3e444c96284a839becc783"
format_stamp: "Formatted at 2024-05-03 14:11:58 on dist-test-slave-v27g"
I20240503 14:11:58.478267 17329 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52038
I20240503 14:11:58.480515 16762 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20240503 14:11:58.482254 17469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:58.482632 16762 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240503 14:11:58.482738 16762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714745515241994-16762-0/minicluster-data/ts-1-root
uuid: "09f6d631be3e444c96284a839becc783"
format_stamp: "Formatted at 2024-05-03 14:11:58 on dist-test-slave-v27g"
I20240503 14:11:58.482863 16762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714745515241994-16762-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714745515241994-16762-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714745515241994-16762-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:11:58.494611 16762 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:11:58.495154 16762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:11:58.495702 16762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:11:58.496488 16762 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:11:58.496567 16762 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:58.496665 16762 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:11:58.496740 16762 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:58.523079 16762 rpc_server.cc:307] RPC server started. Bound to: 127.16.94.130:37127
I20240503 14:11:58.523118 17532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.94.130:37127 every 8 connection(s)
I20240503 14:11:58.524744 16762 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:11:58.528439 17537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:11:58.528879 17536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:11:58.529744 16762 server_base.cc:1034] running on GCE node
I20240503 14:11:58.531879 17533 heartbeater.cc:347] Connected to a master server at 127.16.94.190:44937
I20240503 14:11:58.532014 17533 heartbeater.cc:464] Registering TS with master...
I20240503 14:11:58.532384 17533 heartbeater.cc:511] Master 127.16.94.190:44937 requested a full tablet report, sending...
W20240503 14:11:58.532529 17539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:11:58.532946 16762 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:11:58.533020 16762 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:11:58.533103 16762 hybrid_clock.cc:648] HybridClock initialized: now 1714745518533065 us; error 0 us; skew 500 ppm
I20240503 14:11:58.533246 16762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:11:58.533399 17329 ts_manager.cc:194] Registered new tserver with Master: 09f6d631be3e444c96284a839becc783 (127.16.94.130:37127)
I20240503 14:11:58.534395 17329 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52054
I20240503 14:11:58.534687 16762 webserver.cc:460] Webserver started at http://127.16.94.131:44197/ using document root <none> and password file <none>
I20240503 14:11:58.534902 16762 fs_manager.cc:359] Metadata directory not provided
I20240503 14:11:58.535014 16762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:11:58.535163 16762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:11:58.535780 16762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714745515241994-16762-0/minicluster-data/ts-2-root/instance:
uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b"
format_stamp: "Formatted at 2024-05-03 14:11:58 on dist-test-slave-v27g"
I20240503 14:11:58.538141 16762 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 14:11:58.539845 17544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:58.540182 16762 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240503 14:11:58.540292 16762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714745515241994-16762-0/minicluster-data/ts-2-root
uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b"
format_stamp: "Formatted at 2024-05-03 14:11:58 on dist-test-slave-v27g"
I20240503 14:11:58.540426 16762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714745515241994-16762-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714745515241994-16762-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714745515241994-16762-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:11:58.560276 16762 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:11:58.560906 16762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:11:58.561548 16762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:11:58.562465 16762 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:11:58.562536 16762 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:58.562647 16762 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:11:58.562716 16762 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:58.578928 16762 rpc_server.cc:307] RPC server started. Bound to: 127.16.94.131:44333
I20240503 14:11:58.578963 17606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.94.131:44333 every 8 connection(s)
I20240503 14:11:58.585953 17607 heartbeater.cc:347] Connected to a master server at 127.16.94.190:44937
I20240503 14:11:58.586107 17607 heartbeater.cc:464] Registering TS with master...
I20240503 14:11:58.586473 17607 heartbeater.cc:511] Master 127.16.94.190:44937 requested a full tablet report, sending...
I20240503 14:11:58.587294 17329 ts_manager.cc:194] Registered new tserver with Master: 99bfdcfb9ee349ea9eb57b80fc86c71b (127.16.94.131:44333)
I20240503 14:11:58.587342 16762 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.006912404s
I20240503 14:11:58.588052 17329 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52070
I20240503 14:11:59.479504 17457 heartbeater.cc:503] Master 127.16.94.190:44937 was elected leader, sending a full tablet report...
I20240503 14:11:59.535784 17533 heartbeater.cc:503] Master 127.16.94.190:44937 was elected leader, sending a full tablet report...
I20240503 14:11:59.589365 17607 heartbeater.cc:503] Master 127.16.94.190:44937 was elected leader, sending a full tablet report...
I20240503 14:11:59.615262 16762 test_util.cc:268] Using random seed: 1301523720
I20240503 14:11:59.627825 17329 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52086:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\017\004\001\000\376\377\377\037\004\001\000\375\377\377/\004\001\000\374\377\377?\004\001\000\373\377\377O\004\001\000\372\377\377_\004\001\000\371\377\377o""\004\001\000\377\377\377\017\004\001\000\376\377\377\037\004\001\000\375\377\377/\004\001\000\374\377\377?\004\001\000\373\377\377O\004\001\000\372\377\377_\004\001\000\371\377\377o"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:11:59.628603 17329 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:11:59.651891 17572 tablet_service.cc:1467] Processing CreateTablet for tablet 059a1f25723d4ba0be4cbb7185517bed (DEFAULT_TABLE table=test-workload [id=a0b5f416af4d44a691322e0e7f72eab1]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240503 14:11:59.651922 17570 tablet_service.cc:1467] Processing CreateTablet for tablet c89a88f56fb54d10be8eb56f67b3a8ce (DEFAULT_TABLE table=test-workload [id=a0b5f416af4d44a691322e0e7f72eab1]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240503 14:11:59.652659 17569 tablet_service.cc:1467] Processing CreateTablet for tablet 662fbeaef8a64215a1e17b6993c5189b (DEFAULT_TABLE table=test-workload [id=a0b5f416af4d44a691322e0e7f72eab1]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240503 14:11:59.654604 17567 tablet_service.cc:1467] Processing CreateTablet for tablet 4b64a9a4099e47d98fd1534439b1c80c (DEFAULT_TABLE table=test-workload [id=a0b5f416af4d44a691322e0e7f72eab1]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240503 14:11:59.655349 17566 tablet_service.cc:1467] Processing CreateTablet for tablet 9ca0f24b82244f7ebbcb264fd54545bd (DEFAULT_TABLE table=test-workload [id=a0b5f416af4d44a691322e0e7f72eab1]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240503 14:11:59.655890 17569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 662fbeaef8a64215a1e17b6993c5189b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:11:59.656260 17572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 059a1f25723d4ba0be4cbb7185517bed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:11:59.656690 17565 tablet_service.cc:1467] Processing CreateTablet for tablet 121af11b035d49dd8a2f12cc8eb7c583 (DEFAULT_TABLE table=test-workload [id=a0b5f416af4d44a691322e0e7f72eab1]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240503 14:11:59.657193 17565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 121af11b035d49dd8a2f12cc8eb7c583. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:11:59.657867 17568 tablet_service.cc:1467] Processing CreateTablet for tablet f085937f65ad428aa11585f416dd9c54 (DEFAULT_TABLE table=test-workload [id=a0b5f416af4d44a691322e0e7f72eab1]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240503 14:11:59.659168 17571 tablet_service.cc:1467] Processing CreateTablet for tablet ce7190aa241648008185b18f4a5b881f (DEFAULT_TABLE table=test-workload [id=a0b5f416af4d44a691322e0e7f72eab1]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240503 14:11:59.660032 17495 tablet_service.cc:1467] Processing CreateTablet for tablet c89a88f56fb54d10be8eb56f67b3a8ce (DEFAULT_TABLE table=test-workload [id=a0b5f416af4d44a691322e0e7f72eab1]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240503 14:11:59.661002 17495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c89a88f56fb54d10be8eb56f67b3a8ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:11:59.661362 17570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c89a88f56fb54d10be8eb56f67b3a8ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:11:59.662854 17497 tablet_service.cc:1467] Processing CreateTablet for tablet 059a1f25723d4ba0be4cbb7185517bed (DEFAULT_TABLE table=test-workload [id=a0b5f416af4d44a691322e0e7f72eab1]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240503 14:11:59.662088 17496 tablet_service.cc:1467] Processing CreateTablet for tablet ce7190aa241648008185b18f4a5b881f (DEFAULT_TABLE table=test-workload [id=a0b5f416af4d44a691322e0e7f72eab1]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240503 14:11:59.664212 17493 tablet_service.cc:1467] Processing CreateTablet for tablet f085937f65ad428aa11585f416dd9c54 (DEFAULT_TABLE table=test-workload [id=a0b5f416af4d44a691322e0e7f72eab1]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240503 14:11:59.664824 17492 tablet_service.cc:1467] Processing CreateTablet for tablet 4b64a9a4099e47d98fd1534439b1c80c (DEFAULT_TABLE table=test-workload [id=a0b5f416af4d44a691322e0e7f72eab1]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240503 14:11:59.664956 17494 tablet_service.cc:1467] Processing CreateTablet for tablet 662fbeaef8a64215a1e17b6993c5189b (DEFAULT_TABLE table=test-workload [id=a0b5f416af4d44a691322e0e7f72eab1]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240503 14:11:59.665362 17491 tablet_service.cc:1467] Processing CreateTablet for tablet 9ca0f24b82244f7ebbcb264fd54545bd (DEFAULT_TABLE table=test-workload [id=a0b5f416af4d44a691322e0e7f72eab1]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240503 14:11:59.665047 17493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f085937f65ad428aa11585f416dd9c54. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:11:59.666136 17566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ca0f24b82244f7ebbcb264fd54545bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:11:59.666101 17490 tablet_service.cc:1467] Processing CreateTablet for tablet 121af11b035d49dd8a2f12cc8eb7c583 (DEFAULT_TABLE table=test-workload [id=a0b5f416af4d44a691322e0e7f72eab1]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240503 14:11:59.666383 17491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ca0f24b82244f7ebbcb264fd54545bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:11:59.667543 17568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f085937f65ad428aa11585f416dd9c54. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:11:59.670933 17567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b64a9a4099e47d98fd1534439b1c80c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:11:59.674021 17492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b64a9a4099e47d98fd1534439b1c80c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:11:59.675396 17571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce7190aa241648008185b18f4a5b881f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:11:59.681780 17494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 662fbeaef8a64215a1e17b6993c5189b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:11:59.683699 17490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 121af11b035d49dd8a2f12cc8eb7c583. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:11:59.684317 17497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 059a1f25723d4ba0be4cbb7185517bed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:11:59.690390 17496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce7190aa241648008185b18f4a5b881f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:11:59.704461 17626 tablet_bootstrap.cc:492] T 121af11b035d49dd8a2f12cc8eb7c583 P 99bfdcfb9ee349ea9eb57b80fc86c71b: Bootstrap starting.
I20240503 14:11:59.705737 17627 tablet_bootstrap.cc:492] T c89a88f56fb54d10be8eb56f67b3a8ce P 09f6d631be3e444c96284a839becc783: Bootstrap starting.
I20240503 14:11:59.713829 17626 tablet_bootstrap.cc:654] T 121af11b035d49dd8a2f12cc8eb7c583 P 99bfdcfb9ee349ea9eb57b80fc86c71b: Neither blocks nor log segments found. Creating new log.
I20240503 14:11:59.716547 17627 tablet_bootstrap.cc:654] T c89a88f56fb54d10be8eb56f67b3a8ce P 09f6d631be3e444c96284a839becc783: Neither blocks nor log segments found. Creating new log.
I20240503 14:11:59.720562 17420 tablet_service.cc:1467] Processing CreateTablet for tablet c89a88f56fb54d10be8eb56f67b3a8ce (DEFAULT_TABLE table=test-workload [id=a0b5f416af4d44a691322e0e7f72eab1]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240503 14:11:59.720947 17419 tablet_service.cc:1467] Processing CreateTablet for tablet 662fbeaef8a64215a1e17b6993c5189b (DEFAULT_TABLE table=test-workload [id=a0b5f416af4d44a691322e0e7f72eab1]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240503 14:11:59.721177 17420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c89a88f56fb54d10be8eb56f67b3a8ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:11:59.721437 17422 tablet_service.cc:1467] Processing CreateTablet for tablet 059a1f25723d4ba0be4cbb7185517bed (DEFAULT_TABLE table=test-workload [id=a0b5f416af4d44a691322e0e7f72eab1]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240503 14:11:59.721961 17416 tablet_service.cc:1467] Processing CreateTablet for tablet 4b64a9a4099e47d98fd1534439b1c80c (DEFAULT_TABLE table=test-workload [id=a0b5f416af4d44a691322e0e7f72eab1]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240503 14:11:59.722179 17415 tablet_service.cc:1467] Processing CreateTablet for tablet 9ca0f24b82244f7ebbcb264fd54545bd (DEFAULT_TABLE table=test-workload [id=a0b5f416af4d44a691322e0e7f72eab1]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240503 14:11:59.722684 17421 tablet_service.cc:1467] Processing CreateTablet for tablet ce7190aa241648008185b18f4a5b881f (DEFAULT_TABLE table=test-workload [id=a0b5f416af4d44a691322e0e7f72eab1]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240503 14:11:59.722884 17414 tablet_service.cc:1467] Processing CreateTablet for tablet 121af11b035d49dd8a2f12cc8eb7c583 (DEFAULT_TABLE table=test-workload [id=a0b5f416af4d44a691322e0e7f72eab1]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240503 14:11:59.723472 17418 tablet_service.cc:1467] Processing CreateTablet for tablet f085937f65ad428aa11585f416dd9c54 (DEFAULT_TABLE table=test-workload [id=a0b5f416af4d44a691322e0e7f72eab1]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240503 14:11:59.721458 17419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 662fbeaef8a64215a1e17b6993c5189b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:11:59.724303 17421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce7190aa241648008185b18f4a5b881f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:11:59.725764 17414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 121af11b035d49dd8a2f12cc8eb7c583. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:11:59.726033 17422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 059a1f25723d4ba0be4cbb7185517bed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:11:59.726542 17415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ca0f24b82244f7ebbcb264fd54545bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:11:59.727293 17416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b64a9a4099e47d98fd1534439b1c80c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:11:59.730072 17627 tablet_bootstrap.cc:492] T c89a88f56fb54d10be8eb56f67b3a8ce P 09f6d631be3e444c96284a839becc783: No bootstrap required, opened a new log
I20240503 14:11:59.730235 17627 ts_tablet_manager.cc:1367] T c89a88f56fb54d10be8eb56f67b3a8ce P 09f6d631be3e444c96284a839becc783: Time spent bootstrapping tablet: real 0.028s user 0.010s sys 0.000s
I20240503 14:11:59.730863 17626 tablet_bootstrap.cc:492] T 121af11b035d49dd8a2f12cc8eb7c583 P 99bfdcfb9ee349ea9eb57b80fc86c71b: No bootstrap required, opened a new log
I20240503 14:11:59.731117 17626 ts_tablet_manager.cc:1367] T 121af11b035d49dd8a2f12cc8eb7c583 P 99bfdcfb9ee349ea9eb57b80fc86c71b: Time spent bootstrapping tablet: real 0.028s user 0.008s sys 0.004s
I20240503 14:11:59.731231 17627 raft_consensus.cc:348] T c89a88f56fb54d10be8eb56f67b3a8ce P 09f6d631be3e444c96284a839becc783 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } }
I20240503 14:11:59.731487 17627 raft_consensus.cc:374] T c89a88f56fb54d10be8eb56f67b3a8ce P 09f6d631be3e444c96284a839becc783 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:11:59.731588 17627 raft_consensus.cc:729] T c89a88f56fb54d10be8eb56f67b3a8ce P 09f6d631be3e444c96284a839becc783 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09f6d631be3e444c96284a839becc783, State: Initialized, Role: FOLLOWER
I20240503 14:11:59.732234 17626 raft_consensus.cc:348] T 121af11b035d49dd8a2f12cc8eb7c583 P 99bfdcfb9ee349ea9eb57b80fc86c71b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } }
I20240503 14:11:59.732532 17626 raft_consensus.cc:374] T 121af11b035d49dd8a2f12cc8eb7c583 P 99bfdcfb9ee349ea9eb57b80fc86c71b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:11:59.732398 17627 consensus_queue.cc:260] T c89a88f56fb54d10be8eb56f67b3a8ce P 09f6d631be3e444c96284a839becc783 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } }
I20240503 14:11:59.732801 17626 raft_consensus.cc:729] T 121af11b035d49dd8a2f12cc8eb7c583 P 99bfdcfb9ee349ea9eb57b80fc86c71b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99bfdcfb9ee349ea9eb57b80fc86c71b, State: Initialized, Role: FOLLOWER
I20240503 14:11:59.733331 17626 consensus_queue.cc:260] T 121af11b035d49dd8a2f12cc8eb7c583 P 99bfdcfb9ee349ea9eb57b80fc86c71b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } }
I20240503 14:11:59.734905 17418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f085937f65ad428aa11585f416dd9c54. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:11:59.749042 17626 ts_tablet_manager.cc:1398] T 121af11b035d49dd8a2f12cc8eb7c583 P 99bfdcfb9ee349ea9eb57b80fc86c71b: Time spent starting tablet: real 0.018s user 0.008s sys 0.002s
I20240503 14:11:59.749434 17626 tablet_bootstrap.cc:492] T f085937f65ad428aa11585f416dd9c54 P 99bfdcfb9ee349ea9eb57b80fc86c71b: Bootstrap starting.
I20240503 14:11:59.751340 17626 tablet_bootstrap.cc:654] T f085937f65ad428aa11585f416dd9c54 P 99bfdcfb9ee349ea9eb57b80fc86c71b: Neither blocks nor log segments found. Creating new log.
I20240503 14:11:59.751942 17627 ts_tablet_manager.cc:1398] T c89a88f56fb54d10be8eb56f67b3a8ce P 09f6d631be3e444c96284a839becc783: Time spent starting tablet: real 0.022s user 0.016s sys 0.003s
I20240503 14:11:59.752235 17627 tablet_bootstrap.cc:492] T 662fbeaef8a64215a1e17b6993c5189b P 09f6d631be3e444c96284a839becc783: Bootstrap starting.
I20240503 14:11:59.756139 17631 tablet_bootstrap.cc:492] T ce7190aa241648008185b18f4a5b881f P 6ad404efe189443cb1058eaf35888347: Bootstrap starting.
I20240503 14:11:59.758845 17631 tablet_bootstrap.cc:654] T ce7190aa241648008185b18f4a5b881f P 6ad404efe189443cb1058eaf35888347: Neither blocks nor log segments found. Creating new log.
I20240503 14:11:59.761101 17627 tablet_bootstrap.cc:654] T 662fbeaef8a64215a1e17b6993c5189b P 09f6d631be3e444c96284a839becc783: Neither blocks nor log segments found. Creating new log.
I20240503 14:11:59.763391 17626 tablet_bootstrap.cc:492] T f085937f65ad428aa11585f416dd9c54 P 99bfdcfb9ee349ea9eb57b80fc86c71b: No bootstrap required, opened a new log
I20240503 14:11:59.764966 17631 tablet_bootstrap.cc:492] T ce7190aa241648008185b18f4a5b881f P 6ad404efe189443cb1058eaf35888347: No bootstrap required, opened a new log
I20240503 14:11:59.765204 17631 ts_tablet_manager.cc:1367] T ce7190aa241648008185b18f4a5b881f P 6ad404efe189443cb1058eaf35888347: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20240503 14:11:59.766330 17631 raft_consensus.cc:348] T ce7190aa241648008185b18f4a5b881f P 6ad404efe189443cb1058eaf35888347 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } }
I20240503 14:11:59.766616 17631 raft_consensus.cc:374] T ce7190aa241648008185b18f4a5b881f P 6ad404efe189443cb1058eaf35888347 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:11:59.766734 17631 raft_consensus.cc:729] T ce7190aa241648008185b18f4a5b881f P 6ad404efe189443cb1058eaf35888347 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ad404efe189443cb1058eaf35888347, State: Initialized, Role: FOLLOWER
I20240503 14:11:59.767060 17631 consensus_queue.cc:260] T ce7190aa241648008185b18f4a5b881f P 6ad404efe189443cb1058eaf35888347 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } }
I20240503 14:11:59.767551 17626 ts_tablet_manager.cc:1367] T f085937f65ad428aa11585f416dd9c54 P 99bfdcfb9ee349ea9eb57b80fc86c71b: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.007s
I20240503 14:11:59.768621 17626 raft_consensus.cc:348] T f085937f65ad428aa11585f416dd9c54 P 99bfdcfb9ee349ea9eb57b80fc86c71b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } }
I20240503 14:11:59.768922 17626 raft_consensus.cc:374] T f085937f65ad428aa11585f416dd9c54 P 99bfdcfb9ee349ea9eb57b80fc86c71b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:11:59.769038 17626 raft_consensus.cc:729] T f085937f65ad428aa11585f416dd9c54 P 99bfdcfb9ee349ea9eb57b80fc86c71b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99bfdcfb9ee349ea9eb57b80fc86c71b, State: Initialized, Role: FOLLOWER
I20240503 14:11:59.770223 17631 ts_tablet_manager.cc:1398] T ce7190aa241648008185b18f4a5b881f P 6ad404efe189443cb1058eaf35888347: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 14:11:59.770661 17631 tablet_bootstrap.cc:492] T 4b64a9a4099e47d98fd1534439b1c80c P 6ad404efe189443cb1058eaf35888347: Bootstrap starting.
I20240503 14:11:59.773056 17631 tablet_bootstrap.cc:654] T 4b64a9a4099e47d98fd1534439b1c80c P 6ad404efe189443cb1058eaf35888347: Neither blocks nor log segments found. Creating new log.
I20240503 14:11:59.769362 17626 consensus_queue.cc:260] T f085937f65ad428aa11585f416dd9c54 P 99bfdcfb9ee349ea9eb57b80fc86c71b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } }
I20240503 14:11:59.774394 17626 ts_tablet_manager.cc:1398] T f085937f65ad428aa11585f416dd9c54 P 99bfdcfb9ee349ea9eb57b80fc86c71b: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240503 14:11:59.774775 17626 tablet_bootstrap.cc:492] T c89a88f56fb54d10be8eb56f67b3a8ce P 99bfdcfb9ee349ea9eb57b80fc86c71b: Bootstrap starting.
I20240503 14:11:59.776968 17626 tablet_bootstrap.cc:654] T c89a88f56fb54d10be8eb56f67b3a8ce P 99bfdcfb9ee349ea9eb57b80fc86c71b: Neither blocks nor log segments found. Creating new log.
I20240503 14:11:59.779580 17631 tablet_bootstrap.cc:492] T 4b64a9a4099e47d98fd1534439b1c80c P 6ad404efe189443cb1058eaf35888347: No bootstrap required, opened a new log
I20240503 14:11:59.779810 17631 ts_tablet_manager.cc:1367] T 4b64a9a4099e47d98fd1534439b1c80c P 6ad404efe189443cb1058eaf35888347: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20240503 14:11:59.780774 17631 raft_consensus.cc:348] T 4b64a9a4099e47d98fd1534439b1c80c P 6ad404efe189443cb1058eaf35888347 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } }
I20240503 14:11:59.781078 17631 raft_consensus.cc:374] T 4b64a9a4099e47d98fd1534439b1c80c P 6ad404efe189443cb1058eaf35888347 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:11:59.781183 17631 raft_consensus.cc:729] T 4b64a9a4099e47d98fd1534439b1c80c P 6ad404efe189443cb1058eaf35888347 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ad404efe189443cb1058eaf35888347, State: Initialized, Role: FOLLOWER
I20240503 14:11:59.781491 17631 consensus_queue.cc:260] T 4b64a9a4099e47d98fd1534439b1c80c P 6ad404efe189443cb1058eaf35888347 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } }
I20240503 14:11:59.782302 17631 ts_tablet_manager.cc:1398] T 4b64a9a4099e47d98fd1534439b1c80c P 6ad404efe189443cb1058eaf35888347: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 14:11:59.782341 17627 tablet_bootstrap.cc:492] T 662fbeaef8a64215a1e17b6993c5189b P 09f6d631be3e444c96284a839becc783: No bootstrap required, opened a new log
I20240503 14:11:59.782518 17627 ts_tablet_manager.cc:1367] T 662fbeaef8a64215a1e17b6993c5189b P 09f6d631be3e444c96284a839becc783: Time spent bootstrapping tablet: real 0.030s user 0.006s sys 0.012s
I20240503 14:11:59.782670 17631 tablet_bootstrap.cc:492] T 662fbeaef8a64215a1e17b6993c5189b P 6ad404efe189443cb1058eaf35888347: Bootstrap starting.
I20240503 14:11:59.783396 17627 raft_consensus.cc:348] T 662fbeaef8a64215a1e17b6993c5189b P 09f6d631be3e444c96284a839becc783 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } }
I20240503 14:11:59.783644 17627 raft_consensus.cc:374] T 662fbeaef8a64215a1e17b6993c5189b P 09f6d631be3e444c96284a839becc783 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:11:59.783730 17627 raft_consensus.cc:729] T 662fbeaef8a64215a1e17b6993c5189b P 09f6d631be3e444c96284a839becc783 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09f6d631be3e444c96284a839becc783, State: Initialized, Role: FOLLOWER
I20240503 14:11:59.784032 17627 consensus_queue.cc:260] T 662fbeaef8a64215a1e17b6993c5189b P 09f6d631be3e444c96284a839becc783 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } }
I20240503 14:11:59.784641 17627 ts_tablet_manager.cc:1398] T 662fbeaef8a64215a1e17b6993c5189b P 09f6d631be3e444c96284a839becc783: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240503 14:11:59.784955 17627 tablet_bootstrap.cc:492] T 121af11b035d49dd8a2f12cc8eb7c583 P 09f6d631be3e444c96284a839becc783: Bootstrap starting.
I20240503 14:11:59.784961 17631 tablet_bootstrap.cc:654] T 662fbeaef8a64215a1e17b6993c5189b P 6ad404efe189443cb1058eaf35888347: Neither blocks nor log segments found. Creating new log.
I20240503 14:11:59.785391 17626 tablet_bootstrap.cc:492] T c89a88f56fb54d10be8eb56f67b3a8ce P 99bfdcfb9ee349ea9eb57b80fc86c71b: No bootstrap required, opened a new log
I20240503 14:11:59.785526 17626 ts_tablet_manager.cc:1367] T c89a88f56fb54d10be8eb56f67b3a8ce P 99bfdcfb9ee349ea9eb57b80fc86c71b: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.005s
I20240503 14:11:59.786412 17626 raft_consensus.cc:348] T c89a88f56fb54d10be8eb56f67b3a8ce P 99bfdcfb9ee349ea9eb57b80fc86c71b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } }
I20240503 14:11:59.786664 17627 tablet_bootstrap.cc:654] T 121af11b035d49dd8a2f12cc8eb7c583 P 09f6d631be3e444c96284a839becc783: Neither blocks nor log segments found. Creating new log.
I20240503 14:11:59.786695 17626 raft_consensus.cc:374] T c89a88f56fb54d10be8eb56f67b3a8ce P 99bfdcfb9ee349ea9eb57b80fc86c71b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:11:59.786808 17626 raft_consensus.cc:729] T c89a88f56fb54d10be8eb56f67b3a8ce P 99bfdcfb9ee349ea9eb57b80fc86c71b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99bfdcfb9ee349ea9eb57b80fc86c71b, State: Initialized, Role: FOLLOWER
I20240503 14:11:59.787158 17626 consensus_queue.cc:260] T c89a88f56fb54d10be8eb56f67b3a8ce P 99bfdcfb9ee349ea9eb57b80fc86c71b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } }
I20240503 14:11:59.790287 17626 ts_tablet_manager.cc:1398] T c89a88f56fb54d10be8eb56f67b3a8ce P 99bfdcfb9ee349ea9eb57b80fc86c71b: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240503 14:11:59.790717 17626 tablet_bootstrap.cc:492] T 9ca0f24b82244f7ebbcb264fd54545bd P 99bfdcfb9ee349ea9eb57b80fc86c71b: Bootstrap starting.
I20240503 14:11:59.792987 17626 tablet_bootstrap.cc:654] T 9ca0f24b82244f7ebbcb264fd54545bd P 99bfdcfb9ee349ea9eb57b80fc86c71b: Neither blocks nor log segments found. Creating new log.
I20240503 14:11:59.795312 17631 tablet_bootstrap.cc:492] T 662fbeaef8a64215a1e17b6993c5189b P 6ad404efe189443cb1058eaf35888347: No bootstrap required, opened a new log
I20240503 14:11:59.795485 17631 ts_tablet_manager.cc:1367] T 662fbeaef8a64215a1e17b6993c5189b P 6ad404efe189443cb1058eaf35888347: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.004s
I20240503 14:11:59.795508 17627 tablet_bootstrap.cc:492] T 121af11b035d49dd8a2f12cc8eb7c583 P 09f6d631be3e444c96284a839becc783: No bootstrap required, opened a new log
I20240503 14:11:59.795683 17627 ts_tablet_manager.cc:1367] T 121af11b035d49dd8a2f12cc8eb7c583 P 09f6d631be3e444c96284a839becc783: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.003s
I20240503 14:11:59.795989 17626 tablet_bootstrap.cc:492] T 9ca0f24b82244f7ebbcb264fd54545bd P 99bfdcfb9ee349ea9eb57b80fc86c71b: No bootstrap required, opened a new log
I20240503 14:11:59.796149 17626 ts_tablet_manager.cc:1367] T 9ca0f24b82244f7ebbcb264fd54545bd P 99bfdcfb9ee349ea9eb57b80fc86c71b: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20240503 14:11:59.796408 17627 raft_consensus.cc:348] T 121af11b035d49dd8a2f12cc8eb7c583 P 09f6d631be3e444c96284a839becc783 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } }
I20240503 14:11:59.796362 17631 raft_consensus.cc:348] T 662fbeaef8a64215a1e17b6993c5189b P 6ad404efe189443cb1058eaf35888347 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } }
I20240503 14:11:59.796589 17627 raft_consensus.cc:374] T 121af11b035d49dd8a2f12cc8eb7c583 P 09f6d631be3e444c96284a839becc783 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:11:59.796599 17631 raft_consensus.cc:374] T 662fbeaef8a64215a1e17b6993c5189b P 6ad404efe189443cb1058eaf35888347 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:11:59.796727 17627 raft_consensus.cc:729] T 121af11b035d49dd8a2f12cc8eb7c583 P 09f6d631be3e444c96284a839becc783 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09f6d631be3e444c96284a839becc783, State: Initialized, Role: FOLLOWER
I20240503 14:11:59.796765 17631 raft_consensus.cc:729] T 662fbeaef8a64215a1e17b6993c5189b P 6ad404efe189443cb1058eaf35888347 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ad404efe189443cb1058eaf35888347, State: Initialized, Role: FOLLOWER
I20240503 14:11:59.796852 17626 raft_consensus.cc:348] T 9ca0f24b82244f7ebbcb264fd54545bd P 99bfdcfb9ee349ea9eb57b80fc86c71b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } }
I20240503 14:11:59.797081 17626 raft_consensus.cc:374] T 9ca0f24b82244f7ebbcb264fd54545bd P 99bfdcfb9ee349ea9eb57b80fc86c71b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:11:59.796993 17627 consensus_queue.cc:260] T 121af11b035d49dd8a2f12cc8eb7c583 P 09f6d631be3e444c96284a839becc783 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } }
I20240503 14:11:59.797201 17626 raft_consensus.cc:729] T 9ca0f24b82244f7ebbcb264fd54545bd P 99bfdcfb9ee349ea9eb57b80fc86c71b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99bfdcfb9ee349ea9eb57b80fc86c71b, State: Initialized, Role: FOLLOWER
I20240503 14:11:59.797089 17631 consensus_queue.cc:260] T 662fbeaef8a64215a1e17b6993c5189b P 6ad404efe189443cb1058eaf35888347 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } }
I20240503 14:11:59.797446 17626 consensus_queue.cc:260] T 9ca0f24b82244f7ebbcb264fd54545bd P 99bfdcfb9ee349ea9eb57b80fc86c71b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } }
I20240503 14:11:59.797580 17627 ts_tablet_manager.cc:1398] T 121af11b035d49dd8a2f12cc8eb7c583 P 09f6d631be3e444c96284a839becc783: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:11:59.797883 17627 tablet_bootstrap.cc:492] T 9ca0f24b82244f7ebbcb264fd54545bd P 09f6d631be3e444c96284a839becc783: Bootstrap starting.
I20240503 14:11:59.797920 17631 ts_tablet_manager.cc:1398] T 662fbeaef8a64215a1e17b6993c5189b P 6ad404efe189443cb1058eaf35888347: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:11:59.798175 17631 tablet_bootstrap.cc:492] T 121af11b035d49dd8a2f12cc8eb7c583 P 6ad404efe189443cb1058eaf35888347: Bootstrap starting.
I20240503 14:11:59.799749 17627 tablet_bootstrap.cc:654] T 9ca0f24b82244f7ebbcb264fd54545bd P 09f6d631be3e444c96284a839becc783: Neither blocks nor log segments found. Creating new log.
I20240503 14:11:59.800305 17626 ts_tablet_manager.cc:1398] T 9ca0f24b82244f7ebbcb264fd54545bd P 99bfdcfb9ee349ea9eb57b80fc86c71b: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 14:11:59.801985 17626 tablet_bootstrap.cc:492] T 4b64a9a4099e47d98fd1534439b1c80c P 99bfdcfb9ee349ea9eb57b80fc86c71b: Bootstrap starting.
I20240503 14:11:59.802821 17631 tablet_bootstrap.cc:654] T 121af11b035d49dd8a2f12cc8eb7c583 P 6ad404efe189443cb1058eaf35888347: Neither blocks nor log segments found. Creating new log.
I20240503 14:11:59.803565 17626 tablet_bootstrap.cc:654] T 4b64a9a4099e47d98fd1534439b1c80c P 99bfdcfb9ee349ea9eb57b80fc86c71b: Neither blocks nor log segments found. Creating new log.
I20240503 14:11:59.807201 17631 tablet_bootstrap.cc:492] T 121af11b035d49dd8a2f12cc8eb7c583 P 6ad404efe189443cb1058eaf35888347: No bootstrap required, opened a new log
I20240503 14:11:59.807427 17631 ts_tablet_manager.cc:1367] T 121af11b035d49dd8a2f12cc8eb7c583 P 6ad404efe189443cb1058eaf35888347: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20240503 14:11:59.808375 17631 raft_consensus.cc:348] T 121af11b035d49dd8a2f12cc8eb7c583 P 6ad404efe189443cb1058eaf35888347 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } }
I20240503 14:11:59.808679 17627 tablet_bootstrap.cc:492] T 9ca0f24b82244f7ebbcb264fd54545bd P 09f6d631be3e444c96284a839becc783: No bootstrap required, opened a new log
I20240503 14:11:59.808688 17631 raft_consensus.cc:374] T 121af11b035d49dd8a2f12cc8eb7c583 P 6ad404efe189443cb1058eaf35888347 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:11:59.808885 17627 ts_tablet_manager.cc:1367] T 9ca0f24b82244f7ebbcb264fd54545bd P 09f6d631be3e444c96284a839becc783: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.005s
I20240503 14:11:59.808892 17631 raft_consensus.cc:729] T 121af11b035d49dd8a2f12cc8eb7c583 P 6ad404efe189443cb1058eaf35888347 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ad404efe189443cb1058eaf35888347, State: Initialized, Role: FOLLOWER
I20240503 14:11:59.809311 17631 consensus_queue.cc:260] T 121af11b035d49dd8a2f12cc8eb7c583 P 6ad404efe189443cb1058eaf35888347 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } }
I20240503 14:11:59.809442 17627 raft_consensus.cc:348] T 9ca0f24b82244f7ebbcb264fd54545bd P 09f6d631be3e444c96284a839becc783 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } }
I20240503 14:11:59.809629 17627 raft_consensus.cc:374] T 9ca0f24b82244f7ebbcb264fd54545bd P 09f6d631be3e444c96284a839becc783 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:11:59.809732 17627 raft_consensus.cc:729] T 9ca0f24b82244f7ebbcb264fd54545bd P 09f6d631be3e444c96284a839becc783 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09f6d631be3e444c96284a839becc783, State: Initialized, Role: FOLLOWER
I20240503 14:11:59.810097 17631 ts_tablet_manager.cc:1398] T 121af11b035d49dd8a2f12cc8eb7c583 P 6ad404efe189443cb1058eaf35888347: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 14:11:59.809994 17627 consensus_queue.cc:260] T 9ca0f24b82244f7ebbcb264fd54545bd P 09f6d631be3e444c96284a839becc783 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } }
I20240503 14:11:59.810505 17627 ts_tablet_manager.cc:1398] T 9ca0f24b82244f7ebbcb264fd54545bd P 09f6d631be3e444c96284a839becc783: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:11:59.810578 17631 tablet_bootstrap.cc:492] T f085937f65ad428aa11585f416dd9c54 P 6ad404efe189443cb1058eaf35888347: Bootstrap starting.
I20240503 14:11:59.810778 17627 tablet_bootstrap.cc:492] T 4b64a9a4099e47d98fd1534439b1c80c P 09f6d631be3e444c96284a839becc783: Bootstrap starting.
I20240503 14:11:59.812482 17627 tablet_bootstrap.cc:654] T 4b64a9a4099e47d98fd1534439b1c80c P 09f6d631be3e444c96284a839becc783: Neither blocks nor log segments found. Creating new log.
I20240503 14:11:59.812741 17631 tablet_bootstrap.cc:654] T f085937f65ad428aa11585f416dd9c54 P 6ad404efe189443cb1058eaf35888347: Neither blocks nor log segments found. Creating new log.
I20240503 14:11:59.815572 17626 tablet_bootstrap.cc:492] T 4b64a9a4099e47d98fd1534439b1c80c P 99bfdcfb9ee349ea9eb57b80fc86c71b: No bootstrap required, opened a new log
I20240503 14:11:59.815762 17626 ts_tablet_manager.cc:1367] T 4b64a9a4099e47d98fd1534439b1c80c P 99bfdcfb9ee349ea9eb57b80fc86c71b: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.012s
I20240503 14:11:59.817019 17626 raft_consensus.cc:348] T 4b64a9a4099e47d98fd1534439b1c80c P 99bfdcfb9ee349ea9eb57b80fc86c71b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } }
I20240503 14:11:59.817303 17626 raft_consensus.cc:374] T 4b64a9a4099e47d98fd1534439b1c80c P 99bfdcfb9ee349ea9eb57b80fc86c71b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:11:59.817421 17626 raft_consensus.cc:729] T 4b64a9a4099e47d98fd1534439b1c80c P 99bfdcfb9ee349ea9eb57b80fc86c71b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99bfdcfb9ee349ea9eb57b80fc86c71b, State: Initialized, Role: FOLLOWER
I20240503 14:11:59.817765 17626 consensus_queue.cc:260] T 4b64a9a4099e47d98fd1534439b1c80c P 99bfdcfb9ee349ea9eb57b80fc86c71b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } }
I20240503 14:11:59.818495 17626 ts_tablet_manager.cc:1398] T 4b64a9a4099e47d98fd1534439b1c80c P 99bfdcfb9ee349ea9eb57b80fc86c71b: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240503 14:11:59.818782 17626 tablet_bootstrap.cc:492] T 662fbeaef8a64215a1e17b6993c5189b P 99bfdcfb9ee349ea9eb57b80fc86c71b: Bootstrap starting.
I20240503 14:11:59.820477 17626 tablet_bootstrap.cc:654] T 662fbeaef8a64215a1e17b6993c5189b P 99bfdcfb9ee349ea9eb57b80fc86c71b: Neither blocks nor log segments found. Creating new log.
I20240503 14:11:59.822681 17627 tablet_bootstrap.cc:492] T 4b64a9a4099e47d98fd1534439b1c80c P 09f6d631be3e444c96284a839becc783: No bootstrap required, opened a new log
I20240503 14:11:59.822876 17627 ts_tablet_manager.cc:1367] T 4b64a9a4099e47d98fd1534439b1c80c P 09f6d631be3e444c96284a839becc783: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20240503 14:11:59.822921 17626 tablet_bootstrap.cc:492] T 662fbeaef8a64215a1e17b6993c5189b P 99bfdcfb9ee349ea9eb57b80fc86c71b: No bootstrap required, opened a new log
I20240503 14:11:59.823081 17626 ts_tablet_manager.cc:1367] T 662fbeaef8a64215a1e17b6993c5189b P 99bfdcfb9ee349ea9eb57b80fc86c71b: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20240503 14:11:59.823621 17626 raft_consensus.cc:348] T 662fbeaef8a64215a1e17b6993c5189b P 99bfdcfb9ee349ea9eb57b80fc86c71b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } }
I20240503 14:11:59.823830 17626 raft_consensus.cc:374] T 662fbeaef8a64215a1e17b6993c5189b P 99bfdcfb9ee349ea9eb57b80fc86c71b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:11:59.823799 17627 raft_consensus.cc:348] T 4b64a9a4099e47d98fd1534439b1c80c P 09f6d631be3e444c96284a839becc783 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } }
I20240503 14:11:59.823935 17626 raft_consensus.cc:729] T 662fbeaef8a64215a1e17b6993c5189b P 99bfdcfb9ee349ea9eb57b80fc86c71b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99bfdcfb9ee349ea9eb57b80fc86c71b, State: Initialized, Role: FOLLOWER
I20240503 14:11:59.824015 17627 raft_consensus.cc:374] T 4b64a9a4099e47d98fd1534439b1c80c P 09f6d631be3e444c96284a839becc783 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:11:59.824126 17627 raft_consensus.cc:729] T 4b64a9a4099e47d98fd1534439b1c80c P 09f6d631be3e444c96284a839becc783 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09f6d631be3e444c96284a839becc783, State: Initialized, Role: FOLLOWER
I20240503 14:11:59.824229 17626 consensus_queue.cc:260] T 662fbeaef8a64215a1e17b6993c5189b P 99bfdcfb9ee349ea9eb57b80fc86c71b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } }
I20240503 14:11:59.824404 17627 consensus_queue.cc:260] T 4b64a9a4099e47d98fd1534439b1c80c P 09f6d631be3e444c96284a839becc783 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } }
I20240503 14:11:59.824661 17631 tablet_bootstrap.cc:492] T f085937f65ad428aa11585f416dd9c54 P 6ad404efe189443cb1058eaf35888347: No bootstrap required, opened a new log
I20240503 14:11:59.824738 17626 ts_tablet_manager.cc:1398] T 662fbeaef8a64215a1e17b6993c5189b P 99bfdcfb9ee349ea9eb57b80fc86c71b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:11:59.824797 17631 ts_tablet_manager.cc:1367] T f085937f65ad428aa11585f416dd9c54 P 6ad404efe189443cb1058eaf35888347: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.004s
I20240503 14:11:59.824931 17627 ts_tablet_manager.cc:1398] T 4b64a9a4099e47d98fd1534439b1c80c P 09f6d631be3e444c96284a839becc783: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240503 14:11:59.825049 17626 tablet_bootstrap.cc:492] T 059a1f25723d4ba0be4cbb7185517bed P 99bfdcfb9ee349ea9eb57b80fc86c71b: Bootstrap starting.
I20240503 14:11:59.825196 17627 tablet_bootstrap.cc:492] T 059a1f25723d4ba0be4cbb7185517bed P 09f6d631be3e444c96284a839becc783: Bootstrap starting.
I20240503 14:11:59.825520 17631 raft_consensus.cc:348] T f085937f65ad428aa11585f416dd9c54 P 6ad404efe189443cb1058eaf35888347 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } }
I20240503 14:11:59.825728 17631 raft_consensus.cc:374] T f085937f65ad428aa11585f416dd9c54 P 6ad404efe189443cb1058eaf35888347 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:11:59.825825 17631 raft_consensus.cc:729] T f085937f65ad428aa11585f416dd9c54 P 6ad404efe189443cb1058eaf35888347 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ad404efe189443cb1058eaf35888347, State: Initialized, Role: FOLLOWER
I20240503 14:11:59.826099 17631 consensus_queue.cc:260] T f085937f65ad428aa11585f416dd9c54 P 6ad404efe189443cb1058eaf35888347 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } }
I20240503 14:11:59.826768 17626 tablet_bootstrap.cc:654] T 059a1f25723d4ba0be4cbb7185517bed P 99bfdcfb9ee349ea9eb57b80fc86c71b: Neither blocks nor log segments found. Creating new log.
I20240503 14:11:59.826916 17627 tablet_bootstrap.cc:654] T 059a1f25723d4ba0be4cbb7185517bed P 09f6d631be3e444c96284a839becc783: Neither blocks nor log segments found. Creating new log.
I20240503 14:11:59.829066 17631 ts_tablet_manager.cc:1398] T f085937f65ad428aa11585f416dd9c54 P 6ad404efe189443cb1058eaf35888347: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:11:59.830379 17631 tablet_bootstrap.cc:492] T c89a88f56fb54d10be8eb56f67b3a8ce P 6ad404efe189443cb1058eaf35888347: Bootstrap starting.
I20240503 14:11:59.833118 17631 tablet_bootstrap.cc:654] T c89a88f56fb54d10be8eb56f67b3a8ce P 6ad404efe189443cb1058eaf35888347: Neither blocks nor log segments found. Creating new log.
I20240503 14:11:59.834837 17626 tablet_bootstrap.cc:492] T 059a1f25723d4ba0be4cbb7185517bed P 99bfdcfb9ee349ea9eb57b80fc86c71b: No bootstrap required, opened a new log
I20240503 14:11:59.834981 17626 ts_tablet_manager.cc:1367] T 059a1f25723d4ba0be4cbb7185517bed P 99bfdcfb9ee349ea9eb57b80fc86c71b: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20240503 14:11:59.835393 17627 tablet_bootstrap.cc:492] T 059a1f25723d4ba0be4cbb7185517bed P 09f6d631be3e444c96284a839becc783: No bootstrap required, opened a new log
I20240503 14:11:59.835631 17627 ts_tablet_manager.cc:1367] T 059a1f25723d4ba0be4cbb7185517bed P 09f6d631be3e444c96284a839becc783: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.003s
I20240503 14:11:59.835669 17626 raft_consensus.cc:348] T 059a1f25723d4ba0be4cbb7185517bed P 99bfdcfb9ee349ea9eb57b80fc86c71b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } } peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } }
I20240503 14:11:59.835932 17626 raft_consensus.cc:374] T 059a1f25723d4ba0be4cbb7185517bed P 99bfdcfb9ee349ea9eb57b80fc86c71b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:11:59.836036 17626 raft_consensus.cc:729] T 059a1f25723d4ba0be4cbb7185517bed P 99bfdcfb9ee349ea9eb57b80fc86c71b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99bfdcfb9ee349ea9eb57b80fc86c71b, State: Initialized, Role: FOLLOWER
I20240503 14:11:59.836252 17626 consensus_queue.cc:260] T 059a1f25723d4ba0be4cbb7185517bed P 99bfdcfb9ee349ea9eb57b80fc86c71b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } } peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } }
I20240503 14:11:59.836280 17627 raft_consensus.cc:348] T 059a1f25723d4ba0be4cbb7185517bed P 09f6d631be3e444c96284a839becc783 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } } peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } }
I20240503 14:11:59.836504 17627 raft_consensus.cc:374] T 059a1f25723d4ba0be4cbb7185517bed P 09f6d631be3e444c96284a839becc783 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:11:59.836584 17627 raft_consensus.cc:729] T 059a1f25723d4ba0be4cbb7185517bed P 09f6d631be3e444c96284a839becc783 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09f6d631be3e444c96284a839becc783, State: Initialized, Role: FOLLOWER
I20240503 14:11:59.836632 17631 tablet_bootstrap.cc:492] T c89a88f56fb54d10be8eb56f67b3a8ce P 6ad404efe189443cb1058eaf35888347: No bootstrap required, opened a new log
I20240503 14:11:59.836810 17626 ts_tablet_manager.cc:1398] T 059a1f25723d4ba0be4cbb7185517bed P 99bfdcfb9ee349ea9eb57b80fc86c71b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:11:59.836850 17631 ts_tablet_manager.cc:1367] T c89a88f56fb54d10be8eb56f67b3a8ce P 6ad404efe189443cb1058eaf35888347: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240503 14:11:59.836935 17627 consensus_queue.cc:260] T 059a1f25723d4ba0be4cbb7185517bed P 09f6d631be3e444c96284a839becc783 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } } peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } }
I20240503 14:11:59.837198 17626 tablet_bootstrap.cc:492] T ce7190aa241648008185b18f4a5b881f P 99bfdcfb9ee349ea9eb57b80fc86c71b: Bootstrap starting.
I20240503 14:11:59.838291 17627 ts_tablet_manager.cc:1398] T 059a1f25723d4ba0be4cbb7185517bed P 09f6d631be3e444c96284a839becc783: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240503 14:11:59.838590 17627 tablet_bootstrap.cc:492] T ce7190aa241648008185b18f4a5b881f P 09f6d631be3e444c96284a839becc783: Bootstrap starting.
I20240503 14:11:59.839161 17626 tablet_bootstrap.cc:654] T ce7190aa241648008185b18f4a5b881f P 99bfdcfb9ee349ea9eb57b80fc86c71b: Neither blocks nor log segments found. Creating new log.
I20240503 14:11:59.840313 17627 tablet_bootstrap.cc:654] T ce7190aa241648008185b18f4a5b881f P 09f6d631be3e444c96284a839becc783: Neither blocks nor log segments found. Creating new log.
I20240503 14:11:59.840711 17631 raft_consensus.cc:348] T c89a88f56fb54d10be8eb56f67b3a8ce P 6ad404efe189443cb1058eaf35888347 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } }
I20240503 14:11:59.840943 17631 raft_consensus.cc:374] T c89a88f56fb54d10be8eb56f67b3a8ce P 6ad404efe189443cb1058eaf35888347 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:11:59.841198 17630 raft_consensus.cc:482] T 4b64a9a4099e47d98fd1534439b1c80c P 09f6d631be3e444c96284a839becc783 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:11:59.841388 17630 raft_consensus.cc:504] T 4b64a9a4099e47d98fd1534439b1c80c P 09f6d631be3e444c96284a839becc783 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } }
I20240503 14:11:59.842741 17630 leader_election.cc:288] T 4b64a9a4099e47d98fd1534439b1c80c P 09f6d631be3e444c96284a839becc783 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ad404efe189443cb1058eaf35888347 (127.16.94.129:42357), 99bfdcfb9ee349ea9eb57b80fc86c71b (127.16.94.131:44333)
I20240503 14:11:59.849257 17631 raft_consensus.cc:729] T c89a88f56fb54d10be8eb56f67b3a8ce P 6ad404efe189443cb1058eaf35888347 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ad404efe189443cb1058eaf35888347, State: Initialized, Role: FOLLOWER
I20240503 14:11:59.849956 17631 consensus_queue.cc:260] T c89a88f56fb54d10be8eb56f67b3a8ce P 6ad404efe189443cb1058eaf35888347 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } }
I20240503 14:11:59.852540 17631 ts_tablet_manager.cc:1398] T c89a88f56fb54d10be8eb56f67b3a8ce P 6ad404efe189443cb1058eaf35888347: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20240503 14:11:59.852946 17631 tablet_bootstrap.cc:492] T 9ca0f24b82244f7ebbcb264fd54545bd P 6ad404efe189443cb1058eaf35888347: Bootstrap starting.
I20240503 14:11:59.855137 17631 tablet_bootstrap.cc:654] T 9ca0f24b82244f7ebbcb264fd54545bd P 6ad404efe189443cb1058eaf35888347: Neither blocks nor log segments found. Creating new log.
I20240503 14:11:59.860059 17630 raft_consensus.cc:482] T 9ca0f24b82244f7ebbcb264fd54545bd P 09f6d631be3e444c96284a839becc783 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:11:59.860276 17630 raft_consensus.cc:504] T 9ca0f24b82244f7ebbcb264fd54545bd P 09f6d631be3e444c96284a839becc783 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } }
I20240503 14:11:59.861110 17630 leader_election.cc:288] T 9ca0f24b82244f7ebbcb264fd54545bd P 09f6d631be3e444c96284a839becc783 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ad404efe189443cb1058eaf35888347 (127.16.94.129:42357), 99bfdcfb9ee349ea9eb57b80fc86c71b (127.16.94.131:44333)
I20240503 14:11:59.870519 17631 tablet_bootstrap.cc:492] T 9ca0f24b82244f7ebbcb264fd54545bd P 6ad404efe189443cb1058eaf35888347: No bootstrap required, opened a new log
I20240503 14:11:59.870716 17631 ts_tablet_manager.cc:1367] T 9ca0f24b82244f7ebbcb264fd54545bd P 6ad404efe189443cb1058eaf35888347: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.000s
I20240503 14:11:59.871587 17432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b64a9a4099e47d98fd1534439b1c80c" candidate_uuid: "09f6d631be3e444c96284a839becc783" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ad404efe189443cb1058eaf35888347" is_pre_election: true
I20240503 14:11:59.872017 17632 raft_consensus.cc:482] T 9ca0f24b82244f7ebbcb264fd54545bd P 99bfdcfb9ee349ea9eb57b80fc86c71b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:11:59.872270 17632 raft_consensus.cc:504] T 9ca0f24b82244f7ebbcb264fd54545bd P 99bfdcfb9ee349ea9eb57b80fc86c71b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } }
I20240503 14:11:59.873637 17632 leader_election.cc:288] T 9ca0f24b82244f7ebbcb264fd54545bd P 99bfdcfb9ee349ea9eb57b80fc86c71b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ad404efe189443cb1058eaf35888347 (127.16.94.129:42357), 09f6d631be3e444c96284a839becc783 (127.16.94.130:37127)
I20240503 14:11:59.873914 17431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ca0f24b82244f7ebbcb264fd54545bd" candidate_uuid: "09f6d631be3e444c96284a839becc783" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ad404efe189443cb1058eaf35888347" is_pre_election: true
I20240503 14:11:59.872057 17432 raft_consensus.cc:2454] T 4b64a9a4099e47d98fd1534439b1c80c P 6ad404efe189443cb1058eaf35888347 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09f6d631be3e444c96284a839becc783 in term 0.
I20240503 14:11:59.875119 17472 leader_election.cc:302] T 4b64a9a4099e47d98fd1534439b1c80c P 09f6d631be3e444c96284a839becc783 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09f6d631be3e444c96284a839becc783, 6ad404efe189443cb1058eaf35888347; no voters:
I20240503 14:11:59.875458 17630 raft_consensus.cc:2781] T 4b64a9a4099e47d98fd1534439b1c80c P 09f6d631be3e444c96284a839becc783 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:11:59.875566 17630 raft_consensus.cc:482] T 4b64a9a4099e47d98fd1534439b1c80c P 09f6d631be3e444c96284a839becc783 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:11:59.875701 17630 raft_consensus.cc:3037] T 4b64a9a4099e47d98fd1534439b1c80c P 09f6d631be3e444c96284a839becc783 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:11:59.879851 17630 raft_consensus.cc:504] T 4b64a9a4099e47d98fd1534439b1c80c P 09f6d631be3e444c96284a839becc783 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } }
I20240503 14:11:59.880910 17630 leader_election.cc:288] T 4b64a9a4099e47d98fd1534439b1c80c P 09f6d631be3e444c96284a839becc783 [CANDIDATE]: Term 1 election: Requested vote from peers 6ad404efe189443cb1058eaf35888347 (127.16.94.129:42357), 99bfdcfb9ee349ea9eb57b80fc86c71b (127.16.94.131:44333)
I20240503 14:11:59.881733 17432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b64a9a4099e47d98fd1534439b1c80c" candidate_uuid: "09f6d631be3e444c96284a839becc783" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ad404efe189443cb1058eaf35888347"
I20240503 14:11:59.881980 17432 raft_consensus.cc:3037] T 4b64a9a4099e47d98fd1534439b1c80c P 6ad404efe189443cb1058eaf35888347 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:11:59.886595 17432 raft_consensus.cc:2454] T 4b64a9a4099e47d98fd1534439b1c80c P 6ad404efe189443cb1058eaf35888347 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09f6d631be3e444c96284a839becc783 in term 1.
I20240503 14:11:59.887243 17472 leader_election.cc:302] T 4b64a9a4099e47d98fd1534439b1c80c P 09f6d631be3e444c96284a839becc783 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09f6d631be3e444c96284a839becc783, 6ad404efe189443cb1058eaf35888347; no voters:
I20240503 14:11:59.887604 17630 raft_consensus.cc:2781] T 4b64a9a4099e47d98fd1534439b1c80c P 09f6d631be3e444c96284a839becc783 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:11:59.871806 17631 raft_consensus.cc:348] T 9ca0f24b82244f7ebbcb264fd54545bd P 6ad404efe189443cb1058eaf35888347 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } }
I20240503 14:11:59.888162 17631 raft_consensus.cc:374] T 9ca0f24b82244f7ebbcb264fd54545bd P 6ad404efe189443cb1058eaf35888347 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:11:59.888526 17630 raft_consensus.cc:686] T 4b64a9a4099e47d98fd1534439b1c80c P 09f6d631be3e444c96284a839becc783 [term 1 LEADER]: Becoming Leader. State: Replica: 09f6d631be3e444c96284a839becc783, State: Running, Role: LEADER
I20240503 14:11:59.888828 17630 consensus_queue.cc:237] T 4b64a9a4099e47d98fd1534439b1c80c P 09f6d631be3e444c96284a839becc783 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } }
I20240503 14:11:59.890272 17631 raft_consensus.cc:729] T 9ca0f24b82244f7ebbcb264fd54545bd P 6ad404efe189443cb1058eaf35888347 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ad404efe189443cb1058eaf35888347, State: Initialized, Role: FOLLOWER
I20240503 14:11:59.890542 17631 consensus_queue.cc:260] T 9ca0f24b82244f7ebbcb264fd54545bd P 6ad404efe189443cb1058eaf35888347 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } }
I20240503 14:11:59.891175 17631 ts_tablet_manager.cc:1398] T 9ca0f24b82244f7ebbcb264fd54545bd P 6ad404efe189443cb1058eaf35888347: Time spent starting tablet: real 0.020s user 0.000s sys 0.002s
I20240503 14:11:59.891525 17631 tablet_bootstrap.cc:492] T 059a1f25723d4ba0be4cbb7185517bed P 6ad404efe189443cb1058eaf35888347: Bootstrap starting.
I20240503 14:11:59.891745 17431 raft_consensus.cc:2454] T 9ca0f24b82244f7ebbcb264fd54545bd P 6ad404efe189443cb1058eaf35888347 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09f6d631be3e444c96284a839becc783 in term 0.
I20240503 14:11:59.892834 17472 leader_election.cc:302] T 9ca0f24b82244f7ebbcb264fd54545bd P 09f6d631be3e444c96284a839becc783 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09f6d631be3e444c96284a839becc783, 6ad404efe189443cb1058eaf35888347; no voters:
I20240503 14:11:59.893249 17658 raft_consensus.cc:2781] T 9ca0f24b82244f7ebbcb264fd54545bd P 09f6d631be3e444c96284a839becc783 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:11:59.893402 17658 raft_consensus.cc:482] T 9ca0f24b82244f7ebbcb264fd54545bd P 09f6d631be3e444c96284a839becc783 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:11:59.893582 17658 raft_consensus.cc:3037] T 9ca0f24b82244f7ebbcb264fd54545bd P 09f6d631be3e444c96284a839becc783 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:11:59.894336 17631 tablet_bootstrap.cc:654] T 059a1f25723d4ba0be4cbb7185517bed P 6ad404efe189443cb1058eaf35888347: Neither blocks nor log segments found. Creating new log.
I20240503 14:11:59.894397 17582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b64a9a4099e47d98fd1534439b1c80c" candidate_uuid: "09f6d631be3e444c96284a839becc783" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" is_pre_election: true
I20240503 14:11:59.894726 17582 raft_consensus.cc:2454] T 4b64a9a4099e47d98fd1534439b1c80c P 99bfdcfb9ee349ea9eb57b80fc86c71b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09f6d631be3e444c96284a839becc783 in term 0.
I20240503 14:11:59.894678 17580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b64a9a4099e47d98fd1534439b1c80c" candidate_uuid: "09f6d631be3e444c96284a839becc783" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b"
I20240503 14:11:59.894999 17580 raft_consensus.cc:3037] T 4b64a9a4099e47d98fd1534439b1c80c P 99bfdcfb9ee349ea9eb57b80fc86c71b [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:11:59.896145 17626 tablet_bootstrap.cc:492] T ce7190aa241648008185b18f4a5b881f P 99bfdcfb9ee349ea9eb57b80fc86c71b: No bootstrap required, opened a new log
I20240503 14:11:59.896344 17626 ts_tablet_manager.cc:1367] T ce7190aa241648008185b18f4a5b881f P 99bfdcfb9ee349ea9eb57b80fc86c71b: Time spent bootstrapping tablet: real 0.059s user 0.029s sys 0.025s
I20240503 14:11:59.897176 17626 raft_consensus.cc:348] T ce7190aa241648008185b18f4a5b881f P 99bfdcfb9ee349ea9eb57b80fc86c71b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } }
I20240503 14:11:59.897435 17626 raft_consensus.cc:374] T ce7190aa241648008185b18f4a5b881f P 99bfdcfb9ee349ea9eb57b80fc86c71b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:11:59.897531 17626 raft_consensus.cc:729] T ce7190aa241648008185b18f4a5b881f P 99bfdcfb9ee349ea9eb57b80fc86c71b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99bfdcfb9ee349ea9eb57b80fc86c71b, State: Initialized, Role: FOLLOWER
I20240503 14:11:59.897928 17626 consensus_queue.cc:260] T ce7190aa241648008185b18f4a5b881f P 99bfdcfb9ee349ea9eb57b80fc86c71b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } }
I20240503 14:11:59.898522 17626 ts_tablet_manager.cc:1398] T ce7190aa241648008185b18f4a5b881f P 99bfdcfb9ee349ea9eb57b80fc86c71b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:11:59.899621 17580 raft_consensus.cc:2454] T 4b64a9a4099e47d98fd1534439b1c80c P 99bfdcfb9ee349ea9eb57b80fc86c71b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09f6d631be3e444c96284a839becc783 in term 1.
I20240503 14:11:59.903296 17627 tablet_bootstrap.cc:492] T ce7190aa241648008185b18f4a5b881f P 09f6d631be3e444c96284a839becc783: No bootstrap required, opened a new log
I20240503 14:11:59.903496 17627 ts_tablet_manager.cc:1367] T ce7190aa241648008185b18f4a5b881f P 09f6d631be3e444c96284a839becc783: Time spent bootstrapping tablet: real 0.065s user 0.017s sys 0.039s
I20240503 14:11:59.904728 17627 raft_consensus.cc:348] T ce7190aa241648008185b18f4a5b881f P 09f6d631be3e444c96284a839becc783 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } }
I20240503 14:11:59.905026 17627 raft_consensus.cc:374] T ce7190aa241648008185b18f4a5b881f P 09f6d631be3e444c96284a839becc783 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:11:59.905179 17627 raft_consensus.cc:729] T ce7190aa241648008185b18f4a5b881f P 09f6d631be3e444c96284a839becc783 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09f6d631be3e444c96284a839becc783, State: Initialized, Role: FOLLOWER
I20240503 14:11:59.905436 17627 consensus_queue.cc:260] T ce7190aa241648008185b18f4a5b881f P 09f6d631be3e444c96284a839becc783 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } }
I20240503 14:11:59.906318 17627 ts_tablet_manager.cc:1398] T ce7190aa241648008185b18f4a5b881f P 09f6d631be3e444c96284a839becc783: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 14:11:59.906596 17431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ca0f24b82244f7ebbcb264fd54545bd" candidate_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ad404efe189443cb1058eaf35888347" is_pre_election: true
I20240503 14:11:59.906710 17627 tablet_bootstrap.cc:492] T f085937f65ad428aa11585f416dd9c54 P 09f6d631be3e444c96284a839becc783: Bootstrap starting.
I20240503 14:11:59.906810 17431 raft_consensus.cc:2454] T 9ca0f24b82244f7ebbcb264fd54545bd P 6ad404efe189443cb1058eaf35888347 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99bfdcfb9ee349ea9eb57b80fc86c71b in term 0.
I20240503 14:11:59.907503 17547 leader_election.cc:302] T 9ca0f24b82244f7ebbcb264fd54545bd P 99bfdcfb9ee349ea9eb57b80fc86c71b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ad404efe189443cb1058eaf35888347, 99bfdcfb9ee349ea9eb57b80fc86c71b; no voters:
I20240503 14:11:59.907871 17632 raft_consensus.cc:2781] T 9ca0f24b82244f7ebbcb264fd54545bd P 99bfdcfb9ee349ea9eb57b80fc86c71b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:11:59.908011 17632 raft_consensus.cc:482] T 9ca0f24b82244f7ebbcb264fd54545bd P 99bfdcfb9ee349ea9eb57b80fc86c71b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:11:59.908102 17632 raft_consensus.cc:3037] T 9ca0f24b82244f7ebbcb264fd54545bd P 99bfdcfb9ee349ea9eb57b80fc86c71b [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:11:59.909085 17627 tablet_bootstrap.cc:654] T f085937f65ad428aa11585f416dd9c54 P 09f6d631be3e444c96284a839becc783: Neither blocks nor log segments found. Creating new log.
I20240503 14:11:59.910225 17631 tablet_bootstrap.cc:492] T 059a1f25723d4ba0be4cbb7185517bed P 6ad404efe189443cb1058eaf35888347: No bootstrap required, opened a new log
I20240503 14:11:59.910403 17631 ts_tablet_manager.cc:1367] T 059a1f25723d4ba0be4cbb7185517bed P 6ad404efe189443cb1058eaf35888347: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.005s
I20240503 14:11:59.910988 17631 raft_consensus.cc:348] T 059a1f25723d4ba0be4cbb7185517bed P 6ad404efe189443cb1058eaf35888347 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } } peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } }
I20240503 14:11:59.911195 17631 raft_consensus.cc:374] T 059a1f25723d4ba0be4cbb7185517bed P 6ad404efe189443cb1058eaf35888347 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:11:59.910992 17632 raft_consensus.cc:504] T 9ca0f24b82244f7ebbcb264fd54545bd P 99bfdcfb9ee349ea9eb57b80fc86c71b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } }
I20240503 14:11:59.911281 17631 raft_consensus.cc:729] T 059a1f25723d4ba0be4cbb7185517bed P 6ad404efe189443cb1058eaf35888347 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ad404efe189443cb1058eaf35888347, State: Initialized, Role: FOLLOWER
I20240503 14:11:59.911489 17631 consensus_queue.cc:260] T 059a1f25723d4ba0be4cbb7185517bed P 6ad404efe189443cb1058eaf35888347 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } } peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } }
I20240503 14:11:59.911823 17632 leader_election.cc:288] T 9ca0f24b82244f7ebbcb264fd54545bd P 99bfdcfb9ee349ea9eb57b80fc86c71b [CANDIDATE]: Term 1 election: Requested vote from peers 6ad404efe189443cb1058eaf35888347 (127.16.94.129:42357), 09f6d631be3e444c96284a839becc783 (127.16.94.130:37127)
I20240503 14:11:59.911969 17631 ts_tablet_manager.cc:1398] T 059a1f25723d4ba0be4cbb7185517bed P 6ad404efe189443cb1058eaf35888347: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:11:59.912511 17431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ca0f24b82244f7ebbcb264fd54545bd" candidate_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ad404efe189443cb1058eaf35888347"
I20240503 14:11:59.912703 17431 raft_consensus.cc:3037] T 9ca0f24b82244f7ebbcb264fd54545bd P 6ad404efe189443cb1058eaf35888347 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:11:59.904803 17327 catalog_manager.cc:5484] T 4b64a9a4099e47d98fd1534439b1c80c P 09f6d631be3e444c96284a839becc783 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09f6d631be3e444c96284a839becc783 (127.16.94.130). New cstate: current_term: 1 leader_uuid: "09f6d631be3e444c96284a839becc783" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } health_report { overall_health: UNKNOWN } } }
I20240503 14:11:59.912739 17507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ca0f24b82244f7ebbcb264fd54545bd" candidate_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09f6d631be3e444c96284a839becc783" is_pre_election: true
I20240503 14:11:59.895201 17581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ca0f24b82244f7ebbcb264fd54545bd" candidate_uuid: "09f6d631be3e444c96284a839becc783" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" is_pre_election: true
I20240503 14:11:59.916015 17581 raft_consensus.cc:2379] T 9ca0f24b82244f7ebbcb264fd54545bd P 99bfdcfb9ee349ea9eb57b80fc86c71b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 09f6d631be3e444c96284a839becc783 in current term 1: Already voted for candidate 99bfdcfb9ee349ea9eb57b80fc86c71b in this term.
I20240503 14:11:59.914410 17506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ca0f24b82244f7ebbcb264fd54545bd" candidate_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09f6d631be3e444c96284a839becc783"
I20240503 14:11:59.920864 17431 raft_consensus.cc:2454] T 9ca0f24b82244f7ebbcb264fd54545bd P 6ad404efe189443cb1058eaf35888347 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99bfdcfb9ee349ea9eb57b80fc86c71b in term 1.
I20240503 14:11:59.921753 17658 raft_consensus.cc:504] T 9ca0f24b82244f7ebbcb264fd54545bd P 09f6d631be3e444c96284a839becc783 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } }
I20240503 14:11:59.921926 17547 leader_election.cc:302] T 9ca0f24b82244f7ebbcb264fd54545bd P 99bfdcfb9ee349ea9eb57b80fc86c71b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6ad404efe189443cb1058eaf35888347, 99bfdcfb9ee349ea9eb57b80fc86c71b; no voters: 09f6d631be3e444c96284a839becc783
I20240503 14:11:59.922446 17632 raft_consensus.cc:2781] T 9ca0f24b82244f7ebbcb264fd54545bd P 99bfdcfb9ee349ea9eb57b80fc86c71b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:11:59.922530 17658 leader_election.cc:288] T 9ca0f24b82244f7ebbcb264fd54545bd P 09f6d631be3e444c96284a839becc783 [CANDIDATE]: Term 1 election: Requested vote from peers 6ad404efe189443cb1058eaf35888347 (127.16.94.129:42357), 99bfdcfb9ee349ea9eb57b80fc86c71b (127.16.94.131:44333)
I20240503 14:11:59.922945 17627 tablet_bootstrap.cc:492] T f085937f65ad428aa11585f416dd9c54 P 09f6d631be3e444c96284a839becc783: No bootstrap required, opened a new log
I20240503 14:11:59.923003 17581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ca0f24b82244f7ebbcb264fd54545bd" candidate_uuid: "09f6d631be3e444c96284a839becc783" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b"
I20240503 14:11:59.923092 17431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ca0f24b82244f7ebbcb264fd54545bd" candidate_uuid: "09f6d631be3e444c96284a839becc783" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ad404efe189443cb1058eaf35888347"
I20240503 14:11:59.923274 17627 ts_tablet_manager.cc:1367] T f085937f65ad428aa11585f416dd9c54 P 09f6d631be3e444c96284a839becc783: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.003s
I20240503 14:11:59.923391 17632 raft_consensus.cc:686] T 9ca0f24b82244f7ebbcb264fd54545bd P 99bfdcfb9ee349ea9eb57b80fc86c71b [term 1 LEADER]: Becoming Leader. State: Replica: 99bfdcfb9ee349ea9eb57b80fc86c71b, State: Running, Role: LEADER
I20240503 14:11:59.923730 17632 consensus_queue.cc:237] T 9ca0f24b82244f7ebbcb264fd54545bd P 99bfdcfb9ee349ea9eb57b80fc86c71b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } }
I20240503 14:11:59.924157 17627 raft_consensus.cc:348] T f085937f65ad428aa11585f416dd9c54 P 09f6d631be3e444c96284a839becc783 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } }
I20240503 14:11:59.924270 17663 raft_consensus.cc:482] T 662fbeaef8a64215a1e17b6993c5189b P 99bfdcfb9ee349ea9eb57b80fc86c71b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:11:59.924396 17627 raft_consensus.cc:374] T f085937f65ad428aa11585f416dd9c54 P 09f6d631be3e444c96284a839becc783 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:11:59.924516 17627 raft_consensus.cc:729] T f085937f65ad428aa11585f416dd9c54 P 09f6d631be3e444c96284a839becc783 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09f6d631be3e444c96284a839becc783, State: Initialized, Role: FOLLOWER
I20240503 14:11:59.923426 17507 raft_consensus.cc:2379] T 9ca0f24b82244f7ebbcb264fd54545bd P 09f6d631be3e444c96284a839becc783 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99bfdcfb9ee349ea9eb57b80fc86c71b in current term 1: Already voted for candidate 09f6d631be3e444c96284a839becc783 in this term.
I20240503 14:11:59.924814 17627 consensus_queue.cc:260] T f085937f65ad428aa11585f416dd9c54 P 09f6d631be3e444c96284a839becc783 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } }
I20240503 14:11:59.923298 17431 raft_consensus.cc:2379] T 9ca0f24b82244f7ebbcb264fd54545bd P 6ad404efe189443cb1058eaf35888347 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 09f6d631be3e444c96284a839becc783 in current term 1: Already voted for candidate 99bfdcfb9ee349ea9eb57b80fc86c71b in this term.
I20240503 14:11:59.925406 17627 ts_tablet_manager.cc:1398] T f085937f65ad428aa11585f416dd9c54 P 09f6d631be3e444c96284a839becc783: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:11:59.924712 17663 raft_consensus.cc:504] T 662fbeaef8a64215a1e17b6993c5189b P 99bfdcfb9ee349ea9eb57b80fc86c71b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } }
I20240503 14:11:59.926363 17471 leader_election.cc:302] T 9ca0f24b82244f7ebbcb264fd54545bd P 09f6d631be3e444c96284a839becc783 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09f6d631be3e444c96284a839becc783; no voters: 6ad404efe189443cb1058eaf35888347, 99bfdcfb9ee349ea9eb57b80fc86c71b
I20240503 14:11:59.926569 17663 leader_election.cc:288] T 662fbeaef8a64215a1e17b6993c5189b P 99bfdcfb9ee349ea9eb57b80fc86c71b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ad404efe189443cb1058eaf35888347 (127.16.94.129:42357), 09f6d631be3e444c96284a839becc783 (127.16.94.130:37127)
I20240503 14:11:59.926769 17658 raft_consensus.cc:2726] T 9ca0f24b82244f7ebbcb264fd54545bd P 09f6d631be3e444c96284a839becc783 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 14:11:59.926919 17431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "662fbeaef8a64215a1e17b6993c5189b" candidate_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ad404efe189443cb1058eaf35888347" is_pre_election: true
I20240503 14:11:59.927153 17431 raft_consensus.cc:2454] T 662fbeaef8a64215a1e17b6993c5189b P 6ad404efe189443cb1058eaf35888347 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99bfdcfb9ee349ea9eb57b80fc86c71b in term 0.
I20240503 14:11:59.927480 17507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "662fbeaef8a64215a1e17b6993c5189b" candidate_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09f6d631be3e444c96284a839becc783" is_pre_election: true
I20240503 14:11:59.927682 17507 raft_consensus.cc:2454] T 662fbeaef8a64215a1e17b6993c5189b P 09f6d631be3e444c96284a839becc783 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99bfdcfb9ee349ea9eb57b80fc86c71b in term 0.
I20240503 14:11:59.928030 17547 leader_election.cc:302] T 662fbeaef8a64215a1e17b6993c5189b P 99bfdcfb9ee349ea9eb57b80fc86c71b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ad404efe189443cb1058eaf35888347, 99bfdcfb9ee349ea9eb57b80fc86c71b; no voters:
I20240503 14:11:59.928368 17329 catalog_manager.cc:5484] T 9ca0f24b82244f7ebbcb264fd54545bd P 99bfdcfb9ee349ea9eb57b80fc86c71b reported cstate change: term changed from 0 to 1, leader changed from <none> to 99bfdcfb9ee349ea9eb57b80fc86c71b (127.16.94.131). New cstate: current_term: 1 leader_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } health_report { overall_health: HEALTHY } } }
I20240503 14:11:59.928879 17663 raft_consensus.cc:2781] T 662fbeaef8a64215a1e17b6993c5189b P 99bfdcfb9ee349ea9eb57b80fc86c71b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:11:59.929009 17663 raft_consensus.cc:482] T 662fbeaef8a64215a1e17b6993c5189b P 99bfdcfb9ee349ea9eb57b80fc86c71b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:11:59.929075 17663 raft_consensus.cc:3037] T 662fbeaef8a64215a1e17b6993c5189b P 99bfdcfb9ee349ea9eb57b80fc86c71b [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:11:59.931631 17663 raft_consensus.cc:504] T 662fbeaef8a64215a1e17b6993c5189b P 99bfdcfb9ee349ea9eb57b80fc86c71b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } }
I20240503 14:11:59.932111 17663 leader_election.cc:288] T 662fbeaef8a64215a1e17b6993c5189b P 99bfdcfb9ee349ea9eb57b80fc86c71b [CANDIDATE]: Term 1 election: Requested vote from peers 6ad404efe189443cb1058eaf35888347 (127.16.94.129:42357), 09f6d631be3e444c96284a839becc783 (127.16.94.130:37127)
I20240503 14:11:59.932493 17431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "662fbeaef8a64215a1e17b6993c5189b" candidate_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ad404efe189443cb1058eaf35888347"
I20240503 14:11:59.932600 17507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "662fbeaef8a64215a1e17b6993c5189b" candidate_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09f6d631be3e444c96284a839becc783"
I20240503 14:11:59.932685 17431 raft_consensus.cc:3037] T 662fbeaef8a64215a1e17b6993c5189b P 6ad404efe189443cb1058eaf35888347 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:11:59.932785 17507 raft_consensus.cc:3037] T 662fbeaef8a64215a1e17b6993c5189b P 09f6d631be3e444c96284a839becc783 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:11:59.934804 17431 raft_consensus.cc:2454] T 662fbeaef8a64215a1e17b6993c5189b P 6ad404efe189443cb1058eaf35888347 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99bfdcfb9ee349ea9eb57b80fc86c71b in term 1.
I20240503 14:11:59.934849 17507 raft_consensus.cc:2454] T 662fbeaef8a64215a1e17b6993c5189b P 09f6d631be3e444c96284a839becc783 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99bfdcfb9ee349ea9eb57b80fc86c71b in term 1.
I20240503 14:11:59.935186 17547 leader_election.cc:302] T 662fbeaef8a64215a1e17b6993c5189b P 99bfdcfb9ee349ea9eb57b80fc86c71b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ad404efe189443cb1058eaf35888347, 99bfdcfb9ee349ea9eb57b80fc86c71b; no voters:
I20240503 14:11:59.935451 17663 raft_consensus.cc:2781] T 662fbeaef8a64215a1e17b6993c5189b P 99bfdcfb9ee349ea9eb57b80fc86c71b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:11:59.935595 17663 raft_consensus.cc:686] T 662fbeaef8a64215a1e17b6993c5189b P 99bfdcfb9ee349ea9eb57b80fc86c71b [term 1 LEADER]: Becoming Leader. State: Replica: 99bfdcfb9ee349ea9eb57b80fc86c71b, State: Running, Role: LEADER
I20240503 14:11:59.935853 17663 consensus_queue.cc:237] T 662fbeaef8a64215a1e17b6993c5189b P 99bfdcfb9ee349ea9eb57b80fc86c71b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } }
I20240503 14:11:59.938356 17329 catalog_manager.cc:5484] T 662fbeaef8a64215a1e17b6993c5189b P 99bfdcfb9ee349ea9eb57b80fc86c71b reported cstate change: term changed from 0 to 1, leader changed from <none> to 99bfdcfb9ee349ea9eb57b80fc86c71b (127.16.94.131). New cstate: current_term: 1 leader_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } health_report { overall_health: HEALTHY } } }
I20240503 14:11:59.942476 17636 raft_consensus.cc:482] T 059a1f25723d4ba0be4cbb7185517bed P 6ad404efe189443cb1058eaf35888347 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:11:59.942595 17636 raft_consensus.cc:504] T 059a1f25723d4ba0be4cbb7185517bed P 6ad404efe189443cb1058eaf35888347 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } } peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } }
I20240503 14:11:59.943317 17636 leader_election.cc:288] T 059a1f25723d4ba0be4cbb7185517bed P 6ad404efe189443cb1058eaf35888347 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99bfdcfb9ee349ea9eb57b80fc86c71b (127.16.94.131:44333), 09f6d631be3e444c96284a839becc783 (127.16.94.130:37127)
I20240503 14:11:59.951308 17507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059a1f25723d4ba0be4cbb7185517bed" candidate_uuid: "6ad404efe189443cb1058eaf35888347" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09f6d631be3e444c96284a839becc783" is_pre_election: true
I20240503 14:11:59.951573 17507 raft_consensus.cc:2454] T 059a1f25723d4ba0be4cbb7185517bed P 09f6d631be3e444c96284a839becc783 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ad404efe189443cb1058eaf35888347 in term 0.
I20240503 14:11:59.951988 17396 leader_election.cc:302] T 059a1f25723d4ba0be4cbb7185517bed P 6ad404efe189443cb1058eaf35888347 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09f6d631be3e444c96284a839becc783, 6ad404efe189443cb1058eaf35888347; no voters:
I20240503 14:11:59.952275 17636 raft_consensus.cc:2781] T 059a1f25723d4ba0be4cbb7185517bed P 6ad404efe189443cb1058eaf35888347 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:11:59.952370 17636 raft_consensus.cc:482] T 059a1f25723d4ba0be4cbb7185517bed P 6ad404efe189443cb1058eaf35888347 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:11:59.952428 17636 raft_consensus.cc:3037] T 059a1f25723d4ba0be4cbb7185517bed P 6ad404efe189443cb1058eaf35888347 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:11:59.952706 17581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059a1f25723d4ba0be4cbb7185517bed" candidate_uuid: "6ad404efe189443cb1058eaf35888347" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" is_pre_election: true
I20240503 14:11:59.952908 17581 raft_consensus.cc:2454] T 059a1f25723d4ba0be4cbb7185517bed P 99bfdcfb9ee349ea9eb57b80fc86c71b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ad404efe189443cb1058eaf35888347 in term 0.
I20240503 14:11:59.954524 17636 raft_consensus.cc:504] T 059a1f25723d4ba0be4cbb7185517bed P 6ad404efe189443cb1058eaf35888347 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } } peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } }
I20240503 14:11:59.955016 17636 leader_election.cc:288] T 059a1f25723d4ba0be4cbb7185517bed P 6ad404efe189443cb1058eaf35888347 [CANDIDATE]: Term 1 election: Requested vote from peers 99bfdcfb9ee349ea9eb57b80fc86c71b (127.16.94.131:44333), 09f6d631be3e444c96284a839becc783 (127.16.94.130:37127)
I20240503 14:11:59.955420 17507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059a1f25723d4ba0be4cbb7185517bed" candidate_uuid: "6ad404efe189443cb1058eaf35888347" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09f6d631be3e444c96284a839becc783"
I20240503 14:11:59.955452 17581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059a1f25723d4ba0be4cbb7185517bed" candidate_uuid: "6ad404efe189443cb1058eaf35888347" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b"
I20240503 14:11:59.955605 17507 raft_consensus.cc:3037] T 059a1f25723d4ba0be4cbb7185517bed P 09f6d631be3e444c96284a839becc783 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:11:59.955627 17581 raft_consensus.cc:3037] T 059a1f25723d4ba0be4cbb7185517bed P 99bfdcfb9ee349ea9eb57b80fc86c71b [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:11:59.957564 17507 raft_consensus.cc:2454] T 059a1f25723d4ba0be4cbb7185517bed P 09f6d631be3e444c96284a839becc783 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ad404efe189443cb1058eaf35888347 in term 1.
I20240503 14:11:59.957657 17581 raft_consensus.cc:2454] T 059a1f25723d4ba0be4cbb7185517bed P 99bfdcfb9ee349ea9eb57b80fc86c71b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ad404efe189443cb1058eaf35888347 in term 1.
I20240503 14:11:59.957995 17396 leader_election.cc:302] T 059a1f25723d4ba0be4cbb7185517bed P 6ad404efe189443cb1058eaf35888347 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09f6d631be3e444c96284a839becc783, 6ad404efe189443cb1058eaf35888347; no voters:
I20240503 14:11:59.958299 17636 raft_consensus.cc:2781] T 059a1f25723d4ba0be4cbb7185517bed P 6ad404efe189443cb1058eaf35888347 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:11:59.958701 17636 raft_consensus.cc:686] T 059a1f25723d4ba0be4cbb7185517bed P 6ad404efe189443cb1058eaf35888347 [term 1 LEADER]: Becoming Leader. State: Replica: 6ad404efe189443cb1058eaf35888347, State: Running, Role: LEADER
I20240503 14:11:59.958969 17636 consensus_queue.cc:237] T 059a1f25723d4ba0be4cbb7185517bed P 6ad404efe189443cb1058eaf35888347 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } } peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } }
I20240503 14:11:59.961508 17329 catalog_manager.cc:5484] T 059a1f25723d4ba0be4cbb7185517bed P 6ad404efe189443cb1058eaf35888347 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ad404efe189443cb1058eaf35888347 (127.16.94.129). New cstate: current_term: 1 leader_uuid: "6ad404efe189443cb1058eaf35888347" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } health_report { overall_health: HEALTHY } } }
I20240503 14:11:59.965785 17636 raft_consensus.cc:482] T 121af11b035d49dd8a2f12cc8eb7c583 P 6ad404efe189443cb1058eaf35888347 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:11:59.965924 17636 raft_consensus.cc:504] T 121af11b035d49dd8a2f12cc8eb7c583 P 6ad404efe189443cb1058eaf35888347 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } }
I20240503 14:11:59.966464 17636 leader_election.cc:288] T 121af11b035d49dd8a2f12cc8eb7c583 P 6ad404efe189443cb1058eaf35888347 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09f6d631be3e444c96284a839becc783 (127.16.94.130:37127), 99bfdcfb9ee349ea9eb57b80fc86c71b (127.16.94.131:44333)
I20240503 14:11:59.966877 17507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "121af11b035d49dd8a2f12cc8eb7c583" candidate_uuid: "6ad404efe189443cb1058eaf35888347" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09f6d631be3e444c96284a839becc783" is_pre_election: true
I20240503 14:11:59.966840 17581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "121af11b035d49dd8a2f12cc8eb7c583" candidate_uuid: "6ad404efe189443cb1058eaf35888347" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" is_pre_election: true
I20240503 14:11:59.967064 17507 raft_consensus.cc:2454] T 121af11b035d49dd8a2f12cc8eb7c583 P 09f6d631be3e444c96284a839becc783 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ad404efe189443cb1058eaf35888347 in term 0.
I20240503 14:11:59.967067 17581 raft_consensus.cc:2454] T 121af11b035d49dd8a2f12cc8eb7c583 P 99bfdcfb9ee349ea9eb57b80fc86c71b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ad404efe189443cb1058eaf35888347 in term 0.
I20240503 14:11:59.967449 17396 leader_election.cc:302] T 121af11b035d49dd8a2f12cc8eb7c583 P 6ad404efe189443cb1058eaf35888347 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09f6d631be3e444c96284a839becc783, 6ad404efe189443cb1058eaf35888347; no voters:
I20240503 14:11:59.967684 17636 raft_consensus.cc:2781] T 121af11b035d49dd8a2f12cc8eb7c583 P 6ad404efe189443cb1058eaf35888347 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:11:59.967767 17636 raft_consensus.cc:482] T 121af11b035d49dd8a2f12cc8eb7c583 P 6ad404efe189443cb1058eaf35888347 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:11:59.967876 17636 raft_consensus.cc:3037] T 121af11b035d49dd8a2f12cc8eb7c583 P 6ad404efe189443cb1058eaf35888347 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:11:59.969758 17636 raft_consensus.cc:504] T 121af11b035d49dd8a2f12cc8eb7c583 P 6ad404efe189443cb1058eaf35888347 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } }
I20240503 14:11:59.970250 17636 leader_election.cc:288] T 121af11b035d49dd8a2f12cc8eb7c583 P 6ad404efe189443cb1058eaf35888347 [CANDIDATE]: Term 1 election: Requested vote from peers 09f6d631be3e444c96284a839becc783 (127.16.94.130:37127), 99bfdcfb9ee349ea9eb57b80fc86c71b (127.16.94.131:44333)
I20240503 14:11:59.970638 17581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "121af11b035d49dd8a2f12cc8eb7c583" candidate_uuid: "6ad404efe189443cb1058eaf35888347" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b"
I20240503 14:11:59.970659 17507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "121af11b035d49dd8a2f12cc8eb7c583" candidate_uuid: "6ad404efe189443cb1058eaf35888347" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09f6d631be3e444c96284a839becc783"
I20240503 14:11:59.970804 17507 raft_consensus.cc:3037] T 121af11b035d49dd8a2f12cc8eb7c583 P 09f6d631be3e444c96284a839becc783 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:11:59.970803 17581 raft_consensus.cc:3037] T 121af11b035d49dd8a2f12cc8eb7c583 P 99bfdcfb9ee349ea9eb57b80fc86c71b [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:11:59.972777 17581 raft_consensus.cc:2454] T 121af11b035d49dd8a2f12cc8eb7c583 P 99bfdcfb9ee349ea9eb57b80fc86c71b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ad404efe189443cb1058eaf35888347 in term 1.
I20240503 14:11:59.972796 17507 raft_consensus.cc:2454] T 121af11b035d49dd8a2f12cc8eb7c583 P 09f6d631be3e444c96284a839becc783 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ad404efe189443cb1058eaf35888347 in term 1.
I20240503 14:11:59.973165 17395 leader_election.cc:302] T 121af11b035d49dd8a2f12cc8eb7c583 P 6ad404efe189443cb1058eaf35888347 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ad404efe189443cb1058eaf35888347, 99bfdcfb9ee349ea9eb57b80fc86c71b; no voters:
I20240503 14:11:59.973408 17636 raft_consensus.cc:2781] T 121af11b035d49dd8a2f12cc8eb7c583 P 6ad404efe189443cb1058eaf35888347 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:11:59.973560 17636 raft_consensus.cc:686] T 121af11b035d49dd8a2f12cc8eb7c583 P 6ad404efe189443cb1058eaf35888347 [term 1 LEADER]: Becoming Leader. State: Replica: 6ad404efe189443cb1058eaf35888347, State: Running, Role: LEADER
I20240503 14:11:59.973850 17636 consensus_queue.cc:237] T 121af11b035d49dd8a2f12cc8eb7c583 P 6ad404efe189443cb1058eaf35888347 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } }
I20240503 14:11:59.976071 17329 catalog_manager.cc:5484] T 121af11b035d49dd8a2f12cc8eb7c583 P 6ad404efe189443cb1058eaf35888347 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ad404efe189443cb1058eaf35888347 (127.16.94.129). New cstate: current_term: 1 leader_uuid: "6ad404efe189443cb1058eaf35888347" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:00.002792 17658 raft_consensus.cc:482] T c89a88f56fb54d10be8eb56f67b3a8ce P 09f6d631be3e444c96284a839becc783 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:00.003043 17658 raft_consensus.cc:504] T c89a88f56fb54d10be8eb56f67b3a8ce P 09f6d631be3e444c96284a839becc783 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } }
I20240503 14:12:00.003700 17658 leader_election.cc:288] T c89a88f56fb54d10be8eb56f67b3a8ce P 09f6d631be3e444c96284a839becc783 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ad404efe189443cb1058eaf35888347 (127.16.94.129:42357), 99bfdcfb9ee349ea9eb57b80fc86c71b (127.16.94.131:44333)
I20240503 14:12:00.004264 17581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c89a88f56fb54d10be8eb56f67b3a8ce" candidate_uuid: "09f6d631be3e444c96284a839becc783" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" is_pre_election: true
I20240503 14:12:00.004271 17431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c89a88f56fb54d10be8eb56f67b3a8ce" candidate_uuid: "09f6d631be3e444c96284a839becc783" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ad404efe189443cb1058eaf35888347" is_pre_election: true
I20240503 14:12:00.004562 17431 raft_consensus.cc:2454] T c89a88f56fb54d10be8eb56f67b3a8ce P 6ad404efe189443cb1058eaf35888347 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09f6d631be3e444c96284a839becc783 in term 0.
I20240503 14:12:00.004568 17581 raft_consensus.cc:2454] T c89a88f56fb54d10be8eb56f67b3a8ce P 99bfdcfb9ee349ea9eb57b80fc86c71b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09f6d631be3e444c96284a839becc783 in term 0.
I20240503 14:12:00.004967 17471 leader_election.cc:302] T c89a88f56fb54d10be8eb56f67b3a8ce P 09f6d631be3e444c96284a839becc783 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09f6d631be3e444c96284a839becc783, 99bfdcfb9ee349ea9eb57b80fc86c71b; no voters:
I20240503 14:12:00.005240 17658 raft_consensus.cc:2781] T c89a88f56fb54d10be8eb56f67b3a8ce P 09f6d631be3e444c96284a839becc783 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:00.005344 17658 raft_consensus.cc:482] T c89a88f56fb54d10be8eb56f67b3a8ce P 09f6d631be3e444c96284a839becc783 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:00.005443 17658 raft_consensus.cc:3037] T c89a88f56fb54d10be8eb56f67b3a8ce P 09f6d631be3e444c96284a839becc783 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:00.007392 17658 raft_consensus.cc:504] T c89a88f56fb54d10be8eb56f67b3a8ce P 09f6d631be3e444c96284a839becc783 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } }
I20240503 14:12:00.007896 17658 leader_election.cc:288] T c89a88f56fb54d10be8eb56f67b3a8ce P 09f6d631be3e444c96284a839becc783 [CANDIDATE]: Term 1 election: Requested vote from peers 6ad404efe189443cb1058eaf35888347 (127.16.94.129:42357), 99bfdcfb9ee349ea9eb57b80fc86c71b (127.16.94.131:44333)
I20240503 14:12:00.008354 17581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c89a88f56fb54d10be8eb56f67b3a8ce" candidate_uuid: "09f6d631be3e444c96284a839becc783" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b"
I20240503 14:12:00.008363 17431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c89a88f56fb54d10be8eb56f67b3a8ce" candidate_uuid: "09f6d631be3e444c96284a839becc783" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ad404efe189443cb1058eaf35888347"
I20240503 14:12:00.008548 17431 raft_consensus.cc:3037] T c89a88f56fb54d10be8eb56f67b3a8ce P 6ad404efe189443cb1058eaf35888347 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:00.008554 17581 raft_consensus.cc:3037] T c89a88f56fb54d10be8eb56f67b3a8ce P 99bfdcfb9ee349ea9eb57b80fc86c71b [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:00.010604 17431 raft_consensus.cc:2454] T c89a88f56fb54d10be8eb56f67b3a8ce P 6ad404efe189443cb1058eaf35888347 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09f6d631be3e444c96284a839becc783 in term 1.
I20240503 14:12:00.010601 17581 raft_consensus.cc:2454] T c89a88f56fb54d10be8eb56f67b3a8ce P 99bfdcfb9ee349ea9eb57b80fc86c71b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09f6d631be3e444c96284a839becc783 in term 1.
I20240503 14:12:00.010996 17471 leader_election.cc:302] T c89a88f56fb54d10be8eb56f67b3a8ce P 09f6d631be3e444c96284a839becc783 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09f6d631be3e444c96284a839becc783, 99bfdcfb9ee349ea9eb57b80fc86c71b; no voters:
I20240503 14:12:00.011292 17658 raft_consensus.cc:2781] T c89a88f56fb54d10be8eb56f67b3a8ce P 09f6d631be3e444c96284a839becc783 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:00.011410 17658 raft_consensus.cc:686] T c89a88f56fb54d10be8eb56f67b3a8ce P 09f6d631be3e444c96284a839becc783 [term 1 LEADER]: Becoming Leader. State: Replica: 09f6d631be3e444c96284a839becc783, State: Running, Role: LEADER
I20240503 14:12:00.011659 17658 consensus_queue.cc:237] T c89a88f56fb54d10be8eb56f67b3a8ce P 09f6d631be3e444c96284a839becc783 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } }
I20240503 14:12:00.014139 17329 catalog_manager.cc:5484] T c89a88f56fb54d10be8eb56f67b3a8ce P 09f6d631be3e444c96284a839becc783 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09f6d631be3e444c96284a839becc783 (127.16.94.130). New cstate: current_term: 1 leader_uuid: "09f6d631be3e444c96284a839becc783" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:00.125497 17663 raft_consensus.cc:482] T ce7190aa241648008185b18f4a5b881f P 99bfdcfb9ee349ea9eb57b80fc86c71b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:00.125742 17663 raft_consensus.cc:504] T ce7190aa241648008185b18f4a5b881f P 99bfdcfb9ee349ea9eb57b80fc86c71b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } }
I20240503 14:12:00.126418 17663 leader_election.cc:288] T ce7190aa241648008185b18f4a5b881f P 99bfdcfb9ee349ea9eb57b80fc86c71b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ad404efe189443cb1058eaf35888347 (127.16.94.129:42357), 09f6d631be3e444c96284a839becc783 (127.16.94.130:37127)
I20240503 14:12:00.126935 17431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce7190aa241648008185b18f4a5b881f" candidate_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ad404efe189443cb1058eaf35888347" is_pre_election: true
I20240503 14:12:00.126958 17507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce7190aa241648008185b18f4a5b881f" candidate_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09f6d631be3e444c96284a839becc783" is_pre_election: true
I20240503 14:12:00.127167 17431 raft_consensus.cc:2454] T ce7190aa241648008185b18f4a5b881f P 6ad404efe189443cb1058eaf35888347 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99bfdcfb9ee349ea9eb57b80fc86c71b in term 0.
I20240503 14:12:00.127224 17507 raft_consensus.cc:2454] T ce7190aa241648008185b18f4a5b881f P 09f6d631be3e444c96284a839becc783 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99bfdcfb9ee349ea9eb57b80fc86c71b in term 0.
I20240503 14:12:00.127569 17547 leader_election.cc:302] T ce7190aa241648008185b18f4a5b881f P 99bfdcfb9ee349ea9eb57b80fc86c71b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ad404efe189443cb1058eaf35888347, 99bfdcfb9ee349ea9eb57b80fc86c71b; no voters:
I20240503 14:12:00.127866 17663 raft_consensus.cc:2781] T ce7190aa241648008185b18f4a5b881f P 99bfdcfb9ee349ea9eb57b80fc86c71b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:00.127945 17663 raft_consensus.cc:482] T ce7190aa241648008185b18f4a5b881f P 99bfdcfb9ee349ea9eb57b80fc86c71b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:00.128055 17663 raft_consensus.cc:3037] T ce7190aa241648008185b18f4a5b881f P 99bfdcfb9ee349ea9eb57b80fc86c71b [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:00.129918 17663 raft_consensus.cc:504] T ce7190aa241648008185b18f4a5b881f P 99bfdcfb9ee349ea9eb57b80fc86c71b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } }
I20240503 14:12:00.130436 17663 leader_election.cc:288] T ce7190aa241648008185b18f4a5b881f P 99bfdcfb9ee349ea9eb57b80fc86c71b [CANDIDATE]: Term 1 election: Requested vote from peers 6ad404efe189443cb1058eaf35888347 (127.16.94.129:42357), 09f6d631be3e444c96284a839becc783 (127.16.94.130:37127)
I20240503 14:12:00.130889 17431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce7190aa241648008185b18f4a5b881f" candidate_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ad404efe189443cb1058eaf35888347"
I20240503 14:12:00.130942 17507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce7190aa241648008185b18f4a5b881f" candidate_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09f6d631be3e444c96284a839becc783"
I20240503 14:12:00.131104 17431 raft_consensus.cc:3037] T ce7190aa241648008185b18f4a5b881f P 6ad404efe189443cb1058eaf35888347 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:00.131146 17507 raft_consensus.cc:3037] T ce7190aa241648008185b18f4a5b881f P 09f6d631be3e444c96284a839becc783 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:00.133158 17431 raft_consensus.cc:2454] T ce7190aa241648008185b18f4a5b881f P 6ad404efe189443cb1058eaf35888347 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99bfdcfb9ee349ea9eb57b80fc86c71b in term 1.
I20240503 14:12:00.133278 17507 raft_consensus.cc:2454] T ce7190aa241648008185b18f4a5b881f P 09f6d631be3e444c96284a839becc783 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99bfdcfb9ee349ea9eb57b80fc86c71b in term 1.
I20240503 14:12:00.133531 17547 leader_election.cc:302] T ce7190aa241648008185b18f4a5b881f P 99bfdcfb9ee349ea9eb57b80fc86c71b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ad404efe189443cb1058eaf35888347, 99bfdcfb9ee349ea9eb57b80fc86c71b; no voters:
I20240503 14:12:00.133777 17663 raft_consensus.cc:2781] T ce7190aa241648008185b18f4a5b881f P 99bfdcfb9ee349ea9eb57b80fc86c71b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:00.133939 17663 raft_consensus.cc:686] T ce7190aa241648008185b18f4a5b881f P 99bfdcfb9ee349ea9eb57b80fc86c71b [term 1 LEADER]: Becoming Leader. State: Replica: 99bfdcfb9ee349ea9eb57b80fc86c71b, State: Running, Role: LEADER
I20240503 14:12:00.134189 17663 consensus_queue.cc:237] T ce7190aa241648008185b18f4a5b881f P 99bfdcfb9ee349ea9eb57b80fc86c71b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } }
I20240503 14:12:00.136734 17329 catalog_manager.cc:5484] T ce7190aa241648008185b18f4a5b881f P 99bfdcfb9ee349ea9eb57b80fc86c71b reported cstate change: term changed from 0 to 1, leader changed from <none> to 99bfdcfb9ee349ea9eb57b80fc86c71b (127.16.94.131). New cstate: current_term: 1 leader_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:00.198175 17658 raft_consensus.cc:482] T f085937f65ad428aa11585f416dd9c54 P 09f6d631be3e444c96284a839becc783 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:00.198429 17658 raft_consensus.cc:504] T f085937f65ad428aa11585f416dd9c54 P 09f6d631be3e444c96284a839becc783 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } }
I20240503 14:12:00.199088 17658 leader_election.cc:288] T f085937f65ad428aa11585f416dd9c54 P 09f6d631be3e444c96284a839becc783 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ad404efe189443cb1058eaf35888347 (127.16.94.129:42357), 99bfdcfb9ee349ea9eb57b80fc86c71b (127.16.94.131:44333)
I20240503 14:12:00.199609 17431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f085937f65ad428aa11585f416dd9c54" candidate_uuid: "09f6d631be3e444c96284a839becc783" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ad404efe189443cb1058eaf35888347" is_pre_election: true
I20240503 14:12:00.199640 17581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f085937f65ad428aa11585f416dd9c54" candidate_uuid: "09f6d631be3e444c96284a839becc783" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" is_pre_election: true
I20240503 14:12:00.199812 17431 raft_consensus.cc:2454] T f085937f65ad428aa11585f416dd9c54 P 6ad404efe189443cb1058eaf35888347 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09f6d631be3e444c96284a839becc783 in term 0.
I20240503 14:12:00.199836 17581 raft_consensus.cc:2454] T f085937f65ad428aa11585f416dd9c54 P 99bfdcfb9ee349ea9eb57b80fc86c71b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09f6d631be3e444c96284a839becc783 in term 0.
I20240503 14:12:00.200181 17472 leader_election.cc:302] T f085937f65ad428aa11585f416dd9c54 P 09f6d631be3e444c96284a839becc783 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09f6d631be3e444c96284a839becc783, 6ad404efe189443cb1058eaf35888347; no voters:
I20240503 14:12:00.200438 17658 raft_consensus.cc:2781] T f085937f65ad428aa11585f416dd9c54 P 09f6d631be3e444c96284a839becc783 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:00.200529 17658 raft_consensus.cc:482] T f085937f65ad428aa11585f416dd9c54 P 09f6d631be3e444c96284a839becc783 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:00.200644 17658 raft_consensus.cc:3037] T f085937f65ad428aa11585f416dd9c54 P 09f6d631be3e444c96284a839becc783 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:00.202497 17658 raft_consensus.cc:504] T f085937f65ad428aa11585f416dd9c54 P 09f6d631be3e444c96284a839becc783 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } }
I20240503 14:12:00.203070 17658 leader_election.cc:288] T f085937f65ad428aa11585f416dd9c54 P 09f6d631be3e444c96284a839becc783 [CANDIDATE]: Term 1 election: Requested vote from peers 6ad404efe189443cb1058eaf35888347 (127.16.94.129:42357), 99bfdcfb9ee349ea9eb57b80fc86c71b (127.16.94.131:44333)
I20240503 14:12:00.203433 17431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f085937f65ad428aa11585f416dd9c54" candidate_uuid: "09f6d631be3e444c96284a839becc783" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ad404efe189443cb1058eaf35888347"
I20240503 14:12:00.203466 17581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f085937f65ad428aa11585f416dd9c54" candidate_uuid: "09f6d631be3e444c96284a839becc783" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b"
I20240503 14:12:00.203653 17581 raft_consensus.cc:3037] T f085937f65ad428aa11585f416dd9c54 P 99bfdcfb9ee349ea9eb57b80fc86c71b [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:00.203651 17431 raft_consensus.cc:3037] T f085937f65ad428aa11585f416dd9c54 P 6ad404efe189443cb1058eaf35888347 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:00.206557 17431 raft_consensus.cc:2454] T f085937f65ad428aa11585f416dd9c54 P 6ad404efe189443cb1058eaf35888347 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09f6d631be3e444c96284a839becc783 in term 1.
I20240503 14:12:00.206583 17581 raft_consensus.cc:2454] T f085937f65ad428aa11585f416dd9c54 P 99bfdcfb9ee349ea9eb57b80fc86c71b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09f6d631be3e444c96284a839becc783 in term 1.
I20240503 14:12:00.206971 17472 leader_election.cc:302] T f085937f65ad428aa11585f416dd9c54 P 09f6d631be3e444c96284a839becc783 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09f6d631be3e444c96284a839becc783, 6ad404efe189443cb1058eaf35888347; no voters:
I20240503 14:12:00.207259 17658 raft_consensus.cc:2781] T f085937f65ad428aa11585f416dd9c54 P 09f6d631be3e444c96284a839becc783 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:00.207417 17658 raft_consensus.cc:686] T f085937f65ad428aa11585f416dd9c54 P 09f6d631be3e444c96284a839becc783 [term 1 LEADER]: Becoming Leader. State: Replica: 09f6d631be3e444c96284a839becc783, State: Running, Role: LEADER
I20240503 14:12:00.207661 17658 consensus_queue.cc:237] T f085937f65ad428aa11585f416dd9c54 P 09f6d631be3e444c96284a839becc783 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } }
I20240503 14:12:00.210574 17329 catalog_manager.cc:5484] T f085937f65ad428aa11585f416dd9c54 P 09f6d631be3e444c96284a839becc783 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09f6d631be3e444c96284a839becc783 (127.16.94.130). New cstate: current_term: 1 leader_uuid: "09f6d631be3e444c96284a839becc783" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:00.326256 17663 consensus_queue.cc:1035] T 662fbeaef8a64215a1e17b6993c5189b P 99bfdcfb9ee349ea9eb57b80fc86c71b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:00.330842 17663 consensus_queue.cc:1035] T 662fbeaef8a64215a1e17b6993c5189b P 99bfdcfb9ee349ea9eb57b80fc86c71b [LEADER]: Connected to new peer: Peer: permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:00.350430 16762 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 14:12:00.352162 17636 consensus_queue.cc:1035] T 059a1f25723d4ba0be4cbb7185517bed P 6ad404efe189443cb1058eaf35888347 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:12:00.354198 17679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:00.354373 17680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:00.356155 16762 server_base.cc:1034] running on GCE node
W20240503 14:12:00.357030 17682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:00.357443 16762 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:12:00.357620 16762 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:12:00.357774 16762 hybrid_clock.cc:648] HybridClock initialized: now 1714745520357737 us; error 0 us; skew 500 ppm
I20240503 14:12:00.358069 16762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:00.359700 17636 consensus_queue.cc:1035] T 059a1f25723d4ba0be4cbb7185517bed P 6ad404efe189443cb1058eaf35888347 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:00.361327 16762 webserver.cc:460] Webserver started at http://127.16.94.132:42129/ using document root <none> and password file <none>
I20240503 14:12:00.361622 16762 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:00.361737 16762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:00.361928 16762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:00.362941 16762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714745515241994-16762-0/minicluster-data/ts-3-root/instance:
uuid: "f03dcea3c66540f98f36ae3540b4d603"
format_stamp: "Formatted at 2024-05-03 14:12:00 on dist-test-slave-v27g"
I20240503 14:12:00.366261 16762 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20240503 14:12:00.367894 17689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:00.368247 16762 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240503 14:12:00.368347 16762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714745515241994-16762-0/minicluster-data/ts-3-root
uuid: "f03dcea3c66540f98f36ae3540b4d603"
format_stamp: "Formatted at 2024-05-03 14:12:00 on dist-test-slave-v27g"
I20240503 14:12:00.368463 16762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714745515241994-16762-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714745515241994-16762-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714745515241994-16762-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:00.372143 17663 consensus_queue.cc:1035] T 9ca0f24b82244f7ebbcb264fd54545bd P 99bfdcfb9ee349ea9eb57b80fc86c71b [LEADER]: Connected to new peer: Peer: permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:00.375643 17663 consensus_queue.cc:1035] T 9ca0f24b82244f7ebbcb264fd54545bd P 99bfdcfb9ee349ea9eb57b80fc86c71b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:00.385839 16762 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:00.386385 16762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:00.386889 16762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:00.387738 16762 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:00.387815 16762 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:00.387903 16762 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:00.387982 16762 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:00.394898 17382 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:12:00.399793 17382 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:12:00.400163 17382 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:12:00.405166 17658 consensus_queue.cc:1035] T 4b64a9a4099e47d98fd1534439b1c80c P 09f6d631be3e444c96284a839becc783 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:00.413252 16762 rpc_server.cc:307] RPC server started. Bound to: 127.16.94.132:35877
I20240503 14:12:00.413395 17755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.94.132:35877 every 8 connection(s)
I20240503 14:12:00.414709 17668 consensus_queue.cc:1035] T 121af11b035d49dd8a2f12cc8eb7c583 P 6ad404efe189443cb1058eaf35888347 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:00.415691 17630 consensus_queue.cc:1035] T 4b64a9a4099e47d98fd1534439b1c80c P 09f6d631be3e444c96284a839becc783 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:00.422932 17668 consensus_queue.cc:1035] T 121af11b035d49dd8a2f12cc8eb7c583 P 6ad404efe189443cb1058eaf35888347 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:00.438001 17756 heartbeater.cc:347] Connected to a master server at 127.16.94.190:44937
I20240503 14:12:00.438175 17756 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:00.438794 17756 heartbeater.cc:511] Master 127.16.94.190:44937 requested a full tablet report, sending...
I20240503 14:12:00.439546 17327 ts_manager.cc:194] Registered new tserver with Master: f03dcea3c66540f98f36ae3540b4d603 (127.16.94.132:35877)
I20240503 14:12:00.440284 17327 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52102
I20240503 14:12:00.509477 17630 consensus_queue.cc:1035] T c89a88f56fb54d10be8eb56f67b3a8ce P 09f6d631be3e444c96284a839becc783 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:00.513438 17764 consensus_queue.cc:1035] T c89a88f56fb54d10be8eb56f67b3a8ce P 09f6d631be3e444c96284a839becc783 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:00.564602 17632 consensus_queue.cc:1035] T ce7190aa241648008185b18f4a5b881f P 99bfdcfb9ee349ea9eb57b80fc86c71b [LEADER]: Connected to new peer: Peer: permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:00.568710 17695 consensus_queue.cc:1035] T ce7190aa241648008185b18f4a5b881f P 99bfdcfb9ee349ea9eb57b80fc86c71b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:00.782609 17658 consensus_queue.cc:1035] T f085937f65ad428aa11585f416dd9c54 P 09f6d631be3e444c96284a839becc783 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:00.786684 17764 consensus_queue.cc:1035] T f085937f65ad428aa11585f416dd9c54 P 09f6d631be3e444c96284a839becc783 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:01.441452 17756 heartbeater.cc:503] Master 127.16.94.190:44937 was elected leader, sending a full tablet report...
I20240503 14:12:02.400651 17382 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:12:02.402141 17382 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:12:02.402308 17382 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:12:04.402714 17382 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:12:04.404138 17382 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:12:04.404330 17382 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:12:06.404695 17382 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:12:06.407341 17382 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:12:06.407565 17382 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:12:06.408668 17506 consensus_queue.cc:237] T c89a88f56fb54d10be8eb56f67b3a8ce P 09f6d631be3e444c96284a839becc783 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } attrs { replace: true } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } } peers { permanent_uuid: "f03dcea3c66540f98f36ae3540b4d603" member_type: NON_VOTER last_known_addr { host: "127.16.94.132" port: 35877 } attrs { promote: true } }
I20240503 14:12:06.411089 17432 raft_consensus.cc:1261] T c89a88f56fb54d10be8eb56f67b3a8ce P 6ad404efe189443cb1058eaf35888347 [term 1 FOLLOWER]: Refusing update from remote peer 09f6d631be3e444c96284a839becc783: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:12:06.411850 17581 raft_consensus.cc:1261] T c89a88f56fb54d10be8eb56f67b3a8ce P 99bfdcfb9ee349ea9eb57b80fc86c71b [term 1 FOLLOWER]: Refusing update from remote peer 09f6d631be3e444c96284a839becc783: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:12:06.412058 17778 consensus_queue.cc:1035] T c89a88f56fb54d10be8eb56f67b3a8ce P 09f6d631be3e444c96284a839becc783 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:12:06.412565 17772 consensus_queue.cc:1035] T c89a88f56fb54d10be8eb56f67b3a8ce P 09f6d631be3e444c96284a839becc783 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:12:06.416082 17778 raft_consensus.cc:2932] T c89a88f56fb54d10be8eb56f67b3a8ce P 09f6d631be3e444c96284a839becc783 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f03dcea3c66540f98f36ae3540b4d603 (127.16.94.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } attrs { replace: true } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } } peers { permanent_uuid: "f03dcea3c66540f98f36ae3540b4d603" member_type: NON_VOTER last_known_addr { host: "127.16.94.132" port: 35877 } attrs { promote: true } } }
W20240503 14:12:06.422360 17472 consensus_peers.cc:485] T c89a88f56fb54d10be8eb56f67b3a8ce P 09f6d631be3e444c96284a839becc783 -> Peer f03dcea3c66540f98f36ae3540b4d603 (127.16.94.132:35877): Couldn't send request to peer f03dcea3c66540f98f36ae3540b4d603. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c89a88f56fb54d10be8eb56f67b3a8ce. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:06.421998 17581 raft_consensus.cc:2932] T c89a88f56fb54d10be8eb56f67b3a8ce P 99bfdcfb9ee349ea9eb57b80fc86c71b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f03dcea3c66540f98f36ae3540b4d603 (127.16.94.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } attrs { replace: true } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } } peers { permanent_uuid: "f03dcea3c66540f98f36ae3540b4d603" member_type: NON_VOTER last_known_addr { host: "127.16.94.132" port: 35877 } attrs { promote: true } } }
I20240503 14:12:06.426709 17432 raft_consensus.cc:2932] T c89a88f56fb54d10be8eb56f67b3a8ce P 6ad404efe189443cb1058eaf35888347 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f03dcea3c66540f98f36ae3540b4d603 (127.16.94.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } attrs { replace: true } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } } peers { permanent_uuid: "f03dcea3c66540f98f36ae3540b4d603" member_type: NON_VOTER last_known_addr { host: "127.16.94.132" port: 35877 } attrs { promote: true } } }
I20240503 14:12:06.508858 17329 catalog_manager.cc:5484] T c89a88f56fb54d10be8eb56f67b3a8ce P 09f6d631be3e444c96284a839becc783 reported cstate change: config changed from index -1 to 2, NON_VOTER f03dcea3c66540f98f36ae3540b4d603 (127.16.94.132) added. New cstate: current_term: 1 leader_uuid: "09f6d631be3e444c96284a839becc783" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f03dcea3c66540f98f36ae3540b4d603" member_type: NON_VOTER last_known_addr { host: "127.16.94.132" port: 35877 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:06.527068 17432 consensus_queue.cc:237] T 059a1f25723d4ba0be4cbb7185517bed P 6ad404efe189443cb1058eaf35888347 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } attrs { replace: true } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } } peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } } peers { permanent_uuid: "f03dcea3c66540f98f36ae3540b4d603" member_type: NON_VOTER last_known_addr { host: "127.16.94.132" port: 35877 } attrs { promote: true } }
I20240503 14:12:06.529460 17581 raft_consensus.cc:1261] T 059a1f25723d4ba0be4cbb7185517bed P 99bfdcfb9ee349ea9eb57b80fc86c71b [term 1 FOLLOWER]: Refusing update from remote peer 6ad404efe189443cb1058eaf35888347: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:12:06.529891 17506 raft_consensus.cc:1261] T 059a1f25723d4ba0be4cbb7185517bed P 09f6d631be3e444c96284a839becc783 [term 1 FOLLOWER]: Refusing update from remote peer 6ad404efe189443cb1058eaf35888347: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:12:06.530005 17791 consensus_queue.cc:1035] T 059a1f25723d4ba0be4cbb7185517bed P 6ad404efe189443cb1058eaf35888347 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:12:06.530431 17771 consensus_queue.cc:1035] T 059a1f25723d4ba0be4cbb7185517bed P 6ad404efe189443cb1058eaf35888347 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:12:06.533246 17791 raft_consensus.cc:2932] T 059a1f25723d4ba0be4cbb7185517bed P 6ad404efe189443cb1058eaf35888347 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f03dcea3c66540f98f36ae3540b4d603 (127.16.94.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } attrs { replace: true } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } } peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } } peers { permanent_uuid: "f03dcea3c66540f98f36ae3540b4d603" member_type: NON_VOTER last_known_addr { host: "127.16.94.132" port: 35877 } attrs { promote: true } } }
I20240503 14:12:06.533851 17581 raft_consensus.cc:2932] T 059a1f25723d4ba0be4cbb7185517bed P 99bfdcfb9ee349ea9eb57b80fc86c71b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f03dcea3c66540f98f36ae3540b4d603 (127.16.94.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } attrs { replace: true } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } } peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } } peers { permanent_uuid: "f03dcea3c66540f98f36ae3540b4d603" member_type: NON_VOTER last_known_addr { host: "127.16.94.132" port: 35877 } attrs { promote: true } } }
I20240503 14:12:06.534545 17506 raft_consensus.cc:2932] T 059a1f25723d4ba0be4cbb7185517bed P 09f6d631be3e444c96284a839becc783 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f03dcea3c66540f98f36ae3540b4d603 (127.16.94.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } attrs { replace: true } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } } peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } } peers { permanent_uuid: "f03dcea3c66540f98f36ae3540b4d603" member_type: NON_VOTER last_known_addr { host: "127.16.94.132" port: 35877 } attrs { promote: true } } }
W20240503 14:12:06.538039 17396 consensus_peers.cc:485] T 059a1f25723d4ba0be4cbb7185517bed P 6ad404efe189443cb1058eaf35888347 -> Peer f03dcea3c66540f98f36ae3540b4d603 (127.16.94.132:35877): Couldn't send request to peer f03dcea3c66540f98f36ae3540b4d603. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 059a1f25723d4ba0be4cbb7185517bed. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:06.616627 17328 catalog_manager.cc:5484] T 059a1f25723d4ba0be4cbb7185517bed P 09f6d631be3e444c96284a839becc783 reported cstate change: config changed from index -1 to 2, NON_VOTER f03dcea3c66540f98f36ae3540b4d603 (127.16.94.132) added. New cstate: current_term: 1 leader_uuid: "6ad404efe189443cb1058eaf35888347" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } attrs { replace: true } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } } peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } } peers { permanent_uuid: "f03dcea3c66540f98f36ae3540b4d603" member_type: NON_VOTER last_known_addr { host: "127.16.94.132" port: 35877 } attrs { promote: true } } }
I20240503 14:12:06.625473 17581 consensus_queue.cc:237] T ce7190aa241648008185b18f4a5b881f P 99bfdcfb9ee349ea9eb57b80fc86c71b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } attrs { replace: true } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } } peers { permanent_uuid: "f03dcea3c66540f98f36ae3540b4d603" member_type: NON_VOTER last_known_addr { host: "127.16.94.132" port: 35877 } attrs { promote: true } }
I20240503 14:12:06.628408 17506 raft_consensus.cc:1261] T ce7190aa241648008185b18f4a5b881f P 09f6d631be3e444c96284a839becc783 [term 1 FOLLOWER]: Refusing update from remote peer 99bfdcfb9ee349ea9eb57b80fc86c71b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:12:06.628458 17432 raft_consensus.cc:1261] T ce7190aa241648008185b18f4a5b881f P 6ad404efe189443cb1058eaf35888347 [term 1 FOLLOWER]: Refusing update from remote peer 99bfdcfb9ee349ea9eb57b80fc86c71b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:12:06.629117 17798 consensus_queue.cc:1035] T ce7190aa241648008185b18f4a5b881f P 99bfdcfb9ee349ea9eb57b80fc86c71b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:12:06.629426 17773 consensus_queue.cc:1035] T ce7190aa241648008185b18f4a5b881f P 99bfdcfb9ee349ea9eb57b80fc86c71b [LEADER]: Connected to new peer: Peer: permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:12:06.632484 17773 raft_consensus.cc:2932] T ce7190aa241648008185b18f4a5b881f P 99bfdcfb9ee349ea9eb57b80fc86c71b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f03dcea3c66540f98f36ae3540b4d603 (127.16.94.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } attrs { replace: true } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } } peers { permanent_uuid: "f03dcea3c66540f98f36ae3540b4d603" member_type: NON_VOTER last_known_addr { host: "127.16.94.132" port: 35877 } attrs { promote: true } } }
I20240503 14:12:06.636814 17506 raft_consensus.cc:2932] T ce7190aa241648008185b18f4a5b881f P 09f6d631be3e444c96284a839becc783 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f03dcea3c66540f98f36ae3540b4d603 (127.16.94.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } attrs { replace: true } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } } peers { permanent_uuid: "f03dcea3c66540f98f36ae3540b4d603" member_type: NON_VOTER last_known_addr { host: "127.16.94.132" port: 35877 } attrs { promote: true } } }
W20240503 14:12:06.637199 17547 consensus_peers.cc:485] T ce7190aa241648008185b18f4a5b881f P 99bfdcfb9ee349ea9eb57b80fc86c71b -> Peer f03dcea3c66540f98f36ae3540b4d603 (127.16.94.132:35877): Couldn't send request to peer f03dcea3c66540f98f36ae3540b4d603. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ce7190aa241648008185b18f4a5b881f. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:06.637238 17432 raft_consensus.cc:2932] T ce7190aa241648008185b18f4a5b881f P 6ad404efe189443cb1058eaf35888347 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f03dcea3c66540f98f36ae3540b4d603 (127.16.94.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } attrs { replace: true } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } } peers { permanent_uuid: "f03dcea3c66540f98f36ae3540b4d603" member_type: NON_VOTER last_known_addr { host: "127.16.94.132" port: 35877 } attrs { promote: true } } }
I20240503 14:12:06.714154 17327 catalog_manager.cc:5484] T ce7190aa241648008185b18f4a5b881f P 6ad404efe189443cb1058eaf35888347 reported cstate change: config changed from index -1 to 2, NON_VOTER f03dcea3c66540f98f36ae3540b4d603 (127.16.94.132) added. New cstate: current_term: 1 leader_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } attrs { replace: true } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } } peers { permanent_uuid: "f03dcea3c66540f98f36ae3540b4d603" member_type: NON_VOTER last_known_addr { host: "127.16.94.132" port: 35877 } attrs { promote: true } } }
I20240503 14:12:06.716275 17432 consensus_queue.cc:237] T 121af11b035d49dd8a2f12cc8eb7c583 P 6ad404efe189443cb1058eaf35888347 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } attrs { replace: true } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } } peers { permanent_uuid: "f03dcea3c66540f98f36ae3540b4d603" member_type: NON_VOTER last_known_addr { host: "127.16.94.132" port: 35877 } attrs { promote: true } }
I20240503 14:12:06.719080 17506 raft_consensus.cc:1261] T 121af11b035d49dd8a2f12cc8eb7c583 P 09f6d631be3e444c96284a839becc783 [term 1 FOLLOWER]: Refusing update from remote peer 6ad404efe189443cb1058eaf35888347: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:12:06.719240 17581 raft_consensus.cc:1261] T 121af11b035d49dd8a2f12cc8eb7c583 P 99bfdcfb9ee349ea9eb57b80fc86c71b [term 1 FOLLOWER]: Refusing update from remote peer 6ad404efe189443cb1058eaf35888347: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:12:06.719666 17771 consensus_queue.cc:1035] T 121af11b035d49dd8a2f12cc8eb7c583 P 6ad404efe189443cb1058eaf35888347 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:12:06.719995 17795 consensus_queue.cc:1035] T 121af11b035d49dd8a2f12cc8eb7c583 P 6ad404efe189443cb1058eaf35888347 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 14:12:06.720228 17396 consensus_peers.cc:485] T 121af11b035d49dd8a2f12cc8eb7c583 P 6ad404efe189443cb1058eaf35888347 -> Peer f03dcea3c66540f98f36ae3540b4d603 (127.16.94.132:35877): Couldn't send request to peer f03dcea3c66540f98f36ae3540b4d603. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 121af11b035d49dd8a2f12cc8eb7c583. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:06.722630 17791 raft_consensus.cc:2932] T 121af11b035d49dd8a2f12cc8eb7c583 P 6ad404efe189443cb1058eaf35888347 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f03dcea3c66540f98f36ae3540b4d603 (127.16.94.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } attrs { replace: true } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } } peers { permanent_uuid: "f03dcea3c66540f98f36ae3540b4d603" member_type: NON_VOTER last_known_addr { host: "127.16.94.132" port: 35877 } attrs { promote: true } } }
I20240503 14:12:06.723345 17506 raft_consensus.cc:2932] T 121af11b035d49dd8a2f12cc8eb7c583 P 09f6d631be3e444c96284a839becc783 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f03dcea3c66540f98f36ae3540b4d603 (127.16.94.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } attrs { replace: true } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } } peers { permanent_uuid: "f03dcea3c66540f98f36ae3540b4d603" member_type: NON_VOTER last_known_addr { host: "127.16.94.132" port: 35877 } attrs { promote: true } } }
I20240503 14:12:06.723232 17581 raft_consensus.cc:2932] T 121af11b035d49dd8a2f12cc8eb7c583 P 99bfdcfb9ee349ea9eb57b80fc86c71b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f03dcea3c66540f98f36ae3540b4d603 (127.16.94.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } attrs { replace: true } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } } peers { permanent_uuid: "f03dcea3c66540f98f36ae3540b4d603" member_type: NON_VOTER last_known_addr { host: "127.16.94.132" port: 35877 } attrs { promote: true } } }
I20240503 14:12:06.815450 17328 catalog_manager.cc:5484] T 121af11b035d49dd8a2f12cc8eb7c583 P 09f6d631be3e444c96284a839becc783 reported cstate change: config changed from index -1 to 2, NON_VOTER f03dcea3c66540f98f36ae3540b4d603 (127.16.94.132) added. New cstate: current_term: 1 leader_uuid: "6ad404efe189443cb1058eaf35888347" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } attrs { replace: true } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } } peers { permanent_uuid: "f03dcea3c66540f98f36ae3540b4d603" member_type: NON_VOTER last_known_addr { host: "127.16.94.132" port: 35877 } attrs { promote: true } } }
I20240503 14:12:06.855381 17806 ts_tablet_manager.cc:902] T c89a88f56fb54d10be8eb56f67b3a8ce P f03dcea3c66540f98f36ae3540b4d603: Initiating tablet copy from peer 09f6d631be3e444c96284a839becc783 (127.16.94.130:37127)
I20240503 14:12:06.857496 17806 tablet_copy_client.cc:308] T c89a88f56fb54d10be8eb56f67b3a8ce P f03dcea3c66540f98f36ae3540b4d603: tablet copy: Beginning tablet copy session from remote peer at address 127.16.94.130:37127
I20240503 14:12:06.864296 17518 tablet_copy_service.cc:140] P 09f6d631be3e444c96284a839becc783: Received BeginTabletCopySession request for tablet c89a88f56fb54d10be8eb56f67b3a8ce from peer f03dcea3c66540f98f36ae3540b4d603 ({username='slave'} at 127.0.0.1:48670)
I20240503 14:12:06.864602 17518 tablet_copy_service.cc:161] P 09f6d631be3e444c96284a839becc783: Beginning new tablet copy session on tablet c89a88f56fb54d10be8eb56f67b3a8ce from peer f03dcea3c66540f98f36ae3540b4d603 at {username='slave'} at 127.0.0.1:48670: session id = f03dcea3c66540f98f36ae3540b4d603-c89a88f56fb54d10be8eb56f67b3a8ce
I20240503 14:12:06.867831 17518 tablet_copy_source_session.cc:205] T c89a88f56fb54d10be8eb56f67b3a8ce P 09f6d631be3e444c96284a839becc783: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:12:06.869781 17806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c89a88f56fb54d10be8eb56f67b3a8ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.876547 17806 tablet_copy_client.cc:783] T c89a88f56fb54d10be8eb56f67b3a8ce P f03dcea3c66540f98f36ae3540b4d603: tablet copy: Starting download of 0 data blocks...
I20240503 14:12:06.876920 17806 tablet_copy_client.cc:647] T c89a88f56fb54d10be8eb56f67b3a8ce P f03dcea3c66540f98f36ae3540b4d603: tablet copy: Starting download of 1 WAL segments...
I20240503 14:12:06.879241 17806 tablet_copy_client.cc:523] T c89a88f56fb54d10be8eb56f67b3a8ce P f03dcea3c66540f98f36ae3540b4d603: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:12:06.913589 17806 tablet_bootstrap.cc:492] T c89a88f56fb54d10be8eb56f67b3a8ce P f03dcea3c66540f98f36ae3540b4d603: Bootstrap starting.
I20240503 14:12:06.921120 17806 tablet_bootstrap.cc:492] T c89a88f56fb54d10be8eb56f67b3a8ce P f03dcea3c66540f98f36ae3540b4d603: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:12:06.921535 17806 tablet_bootstrap.cc:492] T c89a88f56fb54d10be8eb56f67b3a8ce P f03dcea3c66540f98f36ae3540b4d603: Bootstrap complete.
I20240503 14:12:06.921828 17806 ts_tablet_manager.cc:1367] T c89a88f56fb54d10be8eb56f67b3a8ce P f03dcea3c66540f98f36ae3540b4d603: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.003s
I20240503 14:12:06.922538 17806 raft_consensus.cc:348] T c89a88f56fb54d10be8eb56f67b3a8ce P f03dcea3c66540f98f36ae3540b4d603 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } attrs { replace: true } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } } peers { permanent_uuid: "f03dcea3c66540f98f36ae3540b4d603" member_type: NON_VOTER last_known_addr { host: "127.16.94.132" port: 35877 } attrs { promote: true } }
I20240503 14:12:06.922751 17806 raft_consensus.cc:729] T c89a88f56fb54d10be8eb56f67b3a8ce P f03dcea3c66540f98f36ae3540b4d603 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f03dcea3c66540f98f36ae3540b4d603, State: Initialized, Role: LEARNER
I20240503 14:12:06.923024 17806 consensus_queue.cc:260] T c89a88f56fb54d10be8eb56f67b3a8ce P f03dcea3c66540f98f36ae3540b4d603 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } attrs { replace: true } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } } peers { permanent_uuid: "f03dcea3c66540f98f36ae3540b4d603" member_type: NON_VOTER last_known_addr { host: "127.16.94.132" port: 35877 } attrs { promote: true } }
I20240503 14:12:06.923926 17806 ts_tablet_manager.cc:1398] T c89a88f56fb54d10be8eb56f67b3a8ce P f03dcea3c66540f98f36ae3540b4d603: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:06.924574 17518 tablet_copy_service.cc:342] P 09f6d631be3e444c96284a839becc783: Request end of tablet copy session f03dcea3c66540f98f36ae3540b4d603-c89a88f56fb54d10be8eb56f67b3a8ce received from {username='slave'} at 127.0.0.1:48670
I20240503 14:12:06.924767 17518 tablet_copy_service.cc:433] P 09f6d631be3e444c96284a839becc783: ending tablet copy session f03dcea3c66540f98f36ae3540b4d603-c89a88f56fb54d10be8eb56f67b3a8ce on tablet c89a88f56fb54d10be8eb56f67b3a8ce with peer f03dcea3c66540f98f36ae3540b4d603
I20240503 14:12:06.964330 17806 ts_tablet_manager.cc:902] T 059a1f25723d4ba0be4cbb7185517bed P f03dcea3c66540f98f36ae3540b4d603: Initiating tablet copy from peer 6ad404efe189443cb1058eaf35888347 (127.16.94.129:42357)
I20240503 14:12:06.965694 17806 tablet_copy_client.cc:308] T 059a1f25723d4ba0be4cbb7185517bed P f03dcea3c66540f98f36ae3540b4d603: tablet copy: Beginning tablet copy session from remote peer at address 127.16.94.129:42357
I20240503 14:12:06.971730 17442 tablet_copy_service.cc:140] P 6ad404efe189443cb1058eaf35888347: Received BeginTabletCopySession request for tablet 059a1f25723d4ba0be4cbb7185517bed from peer f03dcea3c66540f98f36ae3540b4d603 ({username='slave'} at 127.0.0.1:46748)
I20240503 14:12:06.971935 17442 tablet_copy_service.cc:161] P 6ad404efe189443cb1058eaf35888347: Beginning new tablet copy session on tablet 059a1f25723d4ba0be4cbb7185517bed from peer f03dcea3c66540f98f36ae3540b4d603 at {username='slave'} at 127.0.0.1:46748: session id = f03dcea3c66540f98f36ae3540b4d603-059a1f25723d4ba0be4cbb7185517bed
I20240503 14:12:06.974725 17442 tablet_copy_source_session.cc:205] T 059a1f25723d4ba0be4cbb7185517bed P 6ad404efe189443cb1058eaf35888347: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:12:06.975768 17806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 059a1f25723d4ba0be4cbb7185517bed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.980213 17806 tablet_copy_client.cc:783] T 059a1f25723d4ba0be4cbb7185517bed P f03dcea3c66540f98f36ae3540b4d603: tablet copy: Starting download of 0 data blocks...
I20240503 14:12:06.980535 17806 tablet_copy_client.cc:647] T 059a1f25723d4ba0be4cbb7185517bed P f03dcea3c66540f98f36ae3540b4d603: tablet copy: Starting download of 1 WAL segments...
I20240503 14:12:06.981968 17806 tablet_copy_client.cc:523] T 059a1f25723d4ba0be4cbb7185517bed P f03dcea3c66540f98f36ae3540b4d603: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:12:07.012372 17806 tablet_bootstrap.cc:492] T 059a1f25723d4ba0be4cbb7185517bed P f03dcea3c66540f98f36ae3540b4d603: Bootstrap starting.
I20240503 14:12:07.019148 17806 tablet_bootstrap.cc:492] T 059a1f25723d4ba0be4cbb7185517bed P f03dcea3c66540f98f36ae3540b4d603: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:12:07.019583 17806 tablet_bootstrap.cc:492] T 059a1f25723d4ba0be4cbb7185517bed P f03dcea3c66540f98f36ae3540b4d603: Bootstrap complete.
I20240503 14:12:07.019855 17806 ts_tablet_manager.cc:1367] T 059a1f25723d4ba0be4cbb7185517bed P f03dcea3c66540f98f36ae3540b4d603: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20240503 14:12:07.020627 17806 raft_consensus.cc:348] T 059a1f25723d4ba0be4cbb7185517bed P f03dcea3c66540f98f36ae3540b4d603 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } attrs { replace: true } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } } peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } } peers { permanent_uuid: "f03dcea3c66540f98f36ae3540b4d603" member_type: NON_VOTER last_known_addr { host: "127.16.94.132" port: 35877 } attrs { promote: true } }
I20240503 14:12:07.020895 17806 raft_consensus.cc:729] T 059a1f25723d4ba0be4cbb7185517bed P f03dcea3c66540f98f36ae3540b4d603 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f03dcea3c66540f98f36ae3540b4d603, State: Initialized, Role: LEARNER
I20240503 14:12:07.021106 17806 consensus_queue.cc:260] T 059a1f25723d4ba0be4cbb7185517bed P f03dcea3c66540f98f36ae3540b4d603 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } attrs { replace: true } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } } peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } } peers { permanent_uuid: "f03dcea3c66540f98f36ae3540b4d603" member_type: NON_VOTER last_known_addr { host: "127.16.94.132" port: 35877 } attrs { promote: true } }
I20240503 14:12:07.021677 17806 ts_tablet_manager.cc:1398] T 059a1f25723d4ba0be4cbb7185517bed P f03dcea3c66540f98f36ae3540b4d603: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:12:07.022425 17442 tablet_copy_service.cc:342] P 6ad404efe189443cb1058eaf35888347: Request end of tablet copy session f03dcea3c66540f98f36ae3540b4d603-059a1f25723d4ba0be4cbb7185517bed received from {username='slave'} at 127.0.0.1:46748
I20240503 14:12:07.022565 17442 tablet_copy_service.cc:433] P 6ad404efe189443cb1058eaf35888347: ending tablet copy session f03dcea3c66540f98f36ae3540b4d603-059a1f25723d4ba0be4cbb7185517bed on tablet 059a1f25723d4ba0be4cbb7185517bed with peer f03dcea3c66540f98f36ae3540b4d603
I20240503 14:12:07.244460 17806 ts_tablet_manager.cc:902] T ce7190aa241648008185b18f4a5b881f P f03dcea3c66540f98f36ae3540b4d603: Initiating tablet copy from peer 99bfdcfb9ee349ea9eb57b80fc86c71b (127.16.94.131:44333)
I20240503 14:12:07.246351 17806 tablet_copy_client.cc:308] T ce7190aa241648008185b18f4a5b881f P f03dcea3c66540f98f36ae3540b4d603: tablet copy: Beginning tablet copy session from remote peer at address 127.16.94.131:44333
I20240503 14:12:07.253238 17592 tablet_copy_service.cc:140] P 99bfdcfb9ee349ea9eb57b80fc86c71b: Received BeginTabletCopySession request for tablet ce7190aa241648008185b18f4a5b881f from peer f03dcea3c66540f98f36ae3540b4d603 ({username='slave'} at 127.0.0.1:41656)
I20240503 14:12:07.253435 17592 tablet_copy_service.cc:161] P 99bfdcfb9ee349ea9eb57b80fc86c71b: Beginning new tablet copy session on tablet ce7190aa241648008185b18f4a5b881f from peer f03dcea3c66540f98f36ae3540b4d603 at {username='slave'} at 127.0.0.1:41656: session id = f03dcea3c66540f98f36ae3540b4d603-ce7190aa241648008185b18f4a5b881f
I20240503 14:12:07.255939 17592 tablet_copy_source_session.cc:205] T ce7190aa241648008185b18f4a5b881f P 99bfdcfb9ee349ea9eb57b80fc86c71b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:12:07.257180 17806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce7190aa241648008185b18f4a5b881f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:07.262629 17806 tablet_copy_client.cc:783] T ce7190aa241648008185b18f4a5b881f P f03dcea3c66540f98f36ae3540b4d603: tablet copy: Starting download of 0 data blocks...
I20240503 14:12:07.262914 17806 tablet_copy_client.cc:647] T ce7190aa241648008185b18f4a5b881f P f03dcea3c66540f98f36ae3540b4d603: tablet copy: Starting download of 1 WAL segments...
I20240503 14:12:07.264901 17806 tablet_copy_client.cc:523] T ce7190aa241648008185b18f4a5b881f P f03dcea3c66540f98f36ae3540b4d603: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:12:07.313066 17806 tablet_bootstrap.cc:492] T ce7190aa241648008185b18f4a5b881f P f03dcea3c66540f98f36ae3540b4d603: Bootstrap starting.
I20240503 14:12:07.319478 17806 tablet_bootstrap.cc:492] T ce7190aa241648008185b18f4a5b881f P f03dcea3c66540f98f36ae3540b4d603: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:12:07.320024 17806 tablet_bootstrap.cc:492] T ce7190aa241648008185b18f4a5b881f P f03dcea3c66540f98f36ae3540b4d603: Bootstrap complete.
I20240503 14:12:07.320266 17806 ts_tablet_manager.cc:1367] T ce7190aa241648008185b18f4a5b881f P f03dcea3c66540f98f36ae3540b4d603: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.001s
I20240503 14:12:07.321065 17806 raft_consensus.cc:348] T ce7190aa241648008185b18f4a5b881f P f03dcea3c66540f98f36ae3540b4d603 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } attrs { replace: true } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } } peers { permanent_uuid: "f03dcea3c66540f98f36ae3540b4d603" member_type: NON_VOTER last_known_addr { host: "127.16.94.132" port: 35877 } attrs { promote: true } }
I20240503 14:12:07.321288 17806 raft_consensus.cc:729] T ce7190aa241648008185b18f4a5b881f P f03dcea3c66540f98f36ae3540b4d603 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f03dcea3c66540f98f36ae3540b4d603, State: Initialized, Role: LEARNER
I20240503 14:12:07.321511 17806 consensus_queue.cc:260] T ce7190aa241648008185b18f4a5b881f P f03dcea3c66540f98f36ae3540b4d603 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } attrs { replace: true } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } } peers { permanent_uuid: "f03dcea3c66540f98f36ae3540b4d603" member_type: NON_VOTER last_known_addr { host: "127.16.94.132" port: 35877 } attrs { promote: true } }
I20240503 14:12:07.322081 17806 ts_tablet_manager.cc:1398] T ce7190aa241648008185b18f4a5b881f P f03dcea3c66540f98f36ae3540b4d603: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:07.322748 17592 tablet_copy_service.cc:342] P 99bfdcfb9ee349ea9eb57b80fc86c71b: Request end of tablet copy session f03dcea3c66540f98f36ae3540b4d603-ce7190aa241648008185b18f4a5b881f received from {username='slave'} at 127.0.0.1:41656
I20240503 14:12:07.322894 17592 tablet_copy_service.cc:433] P 99bfdcfb9ee349ea9eb57b80fc86c71b: ending tablet copy session f03dcea3c66540f98f36ae3540b4d603-ce7190aa241648008185b18f4a5b881f on tablet ce7190aa241648008185b18f4a5b881f with peer f03dcea3c66540f98f36ae3540b4d603
I20240503 14:12:07.330274 17806 ts_tablet_manager.cc:902] T 121af11b035d49dd8a2f12cc8eb7c583 P f03dcea3c66540f98f36ae3540b4d603: Initiating tablet copy from peer 6ad404efe189443cb1058eaf35888347 (127.16.94.129:42357)
I20240503 14:12:07.331568 17806 tablet_copy_client.cc:308] T 121af11b035d49dd8a2f12cc8eb7c583 P f03dcea3c66540f98f36ae3540b4d603: tablet copy: Beginning tablet copy session from remote peer at address 127.16.94.129:42357
I20240503 14:12:07.332147 17442 tablet_copy_service.cc:140] P 6ad404efe189443cb1058eaf35888347: Received BeginTabletCopySession request for tablet 121af11b035d49dd8a2f12cc8eb7c583 from peer f03dcea3c66540f98f36ae3540b4d603 ({username='slave'} at 127.0.0.1:46748)
I20240503 14:12:07.332338 17442 tablet_copy_service.cc:161] P 6ad404efe189443cb1058eaf35888347: Beginning new tablet copy session on tablet 121af11b035d49dd8a2f12cc8eb7c583 from peer f03dcea3c66540f98f36ae3540b4d603 at {username='slave'} at 127.0.0.1:46748: session id = f03dcea3c66540f98f36ae3540b4d603-121af11b035d49dd8a2f12cc8eb7c583
I20240503 14:12:07.334702 17442 tablet_copy_source_session.cc:205] T 121af11b035d49dd8a2f12cc8eb7c583 P 6ad404efe189443cb1058eaf35888347: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:12:07.335686 17806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 121af11b035d49dd8a2f12cc8eb7c583. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:07.340868 17806 tablet_copy_client.cc:783] T 121af11b035d49dd8a2f12cc8eb7c583 P f03dcea3c66540f98f36ae3540b4d603: tablet copy: Starting download of 0 data blocks...
I20240503 14:12:07.341204 17806 tablet_copy_client.cc:647] T 121af11b035d49dd8a2f12cc8eb7c583 P f03dcea3c66540f98f36ae3540b4d603: tablet copy: Starting download of 1 WAL segments...
I20240503 14:12:07.342723 17806 tablet_copy_client.cc:523] T 121af11b035d49dd8a2f12cc8eb7c583 P f03dcea3c66540f98f36ae3540b4d603: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:12:07.410760 17730 raft_consensus.cc:1203] T c89a88f56fb54d10be8eb56f67b3a8ce P f03dcea3c66540f98f36ae3540b4d603 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:12:07.417800 17806 tablet_bootstrap.cc:492] T 121af11b035d49dd8a2f12cc8eb7c583 P f03dcea3c66540f98f36ae3540b4d603: Bootstrap starting.
I20240503 14:12:07.423228 17806 tablet_bootstrap.cc:492] T 121af11b035d49dd8a2f12cc8eb7c583 P f03dcea3c66540f98f36ae3540b4d603: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:12:07.423645 17806 tablet_bootstrap.cc:492] T 121af11b035d49dd8a2f12cc8eb7c583 P f03dcea3c66540f98f36ae3540b4d603: Bootstrap complete.
I20240503 14:12:07.423913 17806 ts_tablet_manager.cc:1367] T 121af11b035d49dd8a2f12cc8eb7c583 P f03dcea3c66540f98f36ae3540b4d603: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20240503 14:12:07.424605 17806 raft_consensus.cc:348] T 121af11b035d49dd8a2f12cc8eb7c583 P f03dcea3c66540f98f36ae3540b4d603 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } attrs { replace: true } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } } peers { permanent_uuid: "f03dcea3c66540f98f36ae3540b4d603" member_type: NON_VOTER last_known_addr { host: "127.16.94.132" port: 35877 } attrs { promote: true } }
I20240503 14:12:07.424804 17806 raft_consensus.cc:729] T 121af11b035d49dd8a2f12cc8eb7c583 P f03dcea3c66540f98f36ae3540b4d603 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f03dcea3c66540f98f36ae3540b4d603, State: Initialized, Role: LEARNER
I20240503 14:12:07.425034 17806 consensus_queue.cc:260] T 121af11b035d49dd8a2f12cc8eb7c583 P f03dcea3c66540f98f36ae3540b4d603 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ad404efe189443cb1058eaf35888347" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 42357 } } peers { permanent_uuid: "09f6d631be3e444c96284a839becc783" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 37127 } attrs { replace: true } } peers { permanent_uuid: "99bfdcfb9ee349ea9eb57b80fc86c71b" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 44333 } } peers { permanent_uuid: "f03dcea3c66540f98f36ae3540b4d603" member_type: NON_VOTER last_known_addr { host: "127.16.94.132" port: 35877 } attrs { promote: true } }
I20240503 14:12:07.425634 17806 ts_tablet_manager.cc:1398] T 121af11b035d49dd8a2f12cc8eb7c583 P f03dcea3c66540f98f36ae3540b4d603: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:12:07.426252 17442 tablet_copy_service.cc:342] P 6ad404efe189443cb1058eaf35888347: Request end of tablet copy session f03dcea3c66540f98f36ae3540b4d603-121af11b035d49dd8a2f12cc8eb7c583 received from {username='slave'} at 127.0.0.1:46748
I20240503 14:12:07.426358 17442 tablet_copy_service.cc:433] P 6ad404efe189443cb1058eaf35888347: ending tablet copy session f03dcea3c66540f98f36ae3540b4d603-121af11b035d49dd8a2f12cc8eb7c583 on tablet 121af11b035d49dd8a2f12cc8eb7c583 with peer f03dcea3c66540f98f36ae3540b4d603
I20240503 14:12:07.464107 16762 tablet_server.cc:178] TabletServer@127.16.94.129:0 shutting down...
W20240503 14:12:07.469018 17547 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 2 similar messages]
I20240503 14:12:07.473312 16762 ts_tablet_manager.cc:1470] Shutting down tablet manager...
W20240503 14:12:07.473392 17547 consensus_peers.cc:485] T 662fbeaef8a64215a1e17b6993c5189b P 99bfdcfb9ee349ea9eb57b80fc86c71b -> Peer 6ad404efe189443cb1058eaf35888347 (127.16.94.129:42357): Couldn't send request to peer 6ad404efe189443cb1058eaf35888347. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240503 14:12:07.473588 17377 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240503 14:12:07.473713 17377 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240503 14:12:07.473686 16762 tablet_replica.cc:330] T 059a1f25723d4ba0be4cbb7185517bed P 6ad404efe189443cb1058eaf35888347: stopping tablet replica
I20240503 14:12:07.473958 16762 raft_consensus.cc:2229] T 059a1f25723d4ba0be4cbb7185517bed P 6ad404efe189443cb1058eaf35888347 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:07.474426 16762 raft_consensus.cc:2258] T 059a1f25723d4ba0be4cbb7185517bed P 6ad404efe189443cb1058eaf35888347 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:07.475248 16762 tablet_replica.cc:330] T 9ca0f24b82244f7ebbcb264fd54545bd P 6ad404efe189443cb1058eaf35888347: stopping tablet replica
I20240503 14:12:07.475416 16762 raft_consensus.cc:2229] T 9ca0f24b82244f7ebbcb264fd54545bd P 6ad404efe189443cb1058eaf35888347 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:07.475567 16762 raft_consensus.cc:2258] T 9ca0f24b82244f7ebbcb264fd54545bd P 6ad404efe189443cb1058eaf35888347 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:07.476140 16762 tablet_replica.cc:330] T f085937f65ad428aa11585f416dd9c54 P 6ad404efe189443cb1058eaf35888347: stopping tablet replica
I20240503 14:12:07.476310 16762 raft_consensus.cc:2229] T f085937f65ad428aa11585f416dd9c54 P 6ad404efe189443cb1058eaf35888347 [term 1 FOLLOWER]: Raft consensus shutting down.
W20240503 14:12:07.476348 17377 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240503 14:12:07.476416 17377 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240503 14:12:07.476501 16762 raft_consensus.cc:2258] T f085937f65ad428aa11585f416dd9c54 P 6ad404efe189443cb1058eaf35888347 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:07.477011 16762 tablet_replica.cc:330] T 4b64a9a4099e47d98fd1534439b1c80c P 6ad404efe189443cb1058eaf35888347: stopping tablet replica
I20240503 14:12:07.477193 16762 raft_consensus.cc:2229] T 4b64a9a4099e47d98fd1534439b1c80c P 6ad404efe189443cb1058eaf35888347 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:07.477341 16762 raft_consensus.cc:2258] T 4b64a9a4099e47d98fd1534439b1c80c P 6ad404efe189443cb1058eaf35888347 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:07.477916 16762 tablet_replica.cc:330] T ce7190aa241648008185b18f4a5b881f P 6ad404efe189443cb1058eaf35888347: stopping tablet replica
I20240503 14:12:07.478088 16762 raft_consensus.cc:2229] T ce7190aa241648008185b18f4a5b881f P 6ad404efe189443cb1058eaf35888347 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:07.478289 16762 raft_consensus.cc:2258] T ce7190aa241648008185b18f4a5b881f P 6ad404efe189443cb1058eaf35888347 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:07.478893 16762 tablet_replica.cc:330] T 662fbeaef8a64215a1e17b6993c5189b P 6ad404efe189443cb1058eaf35888347: stopping tablet replica
I20240503 14:12:07.479070 16762 raft_consensus.cc:2229] T 662fbeaef8a64215a1e17b6993c5189b P 6ad404efe189443cb1058eaf35888347 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:07.479256 16762 raft_consensus.cc:2258] T 662fbeaef8a64215a1e17b6993c5189b P 6ad404efe189443cb1058eaf35888347 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:07.479763 16762 tablet_replica.cc:330] T 121af11b035d49dd8a2f12cc8eb7c583 P 6ad404efe189443cb1058eaf35888347: stopping tablet replica
I20240503 14:12:07.479911 16762 raft_consensus.cc:2229] T 121af11b035d49dd8a2f12cc8eb7c583 P 6ad404efe189443cb1058eaf35888347 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:07.480216 16762 raft_consensus.cc:2258] T 121af11b035d49dd8a2f12cc8eb7c583 P 6ad404efe189443cb1058eaf35888347 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:07.480868 16762 tablet_replica.cc:330] T c89a88f56fb54d10be8eb56f67b3a8ce P 6ad404efe189443cb1058eaf35888347: stopping tablet replica
I20240503 14:12:07.481007 16762 raft_consensus.cc:2229] T c89a88f56fb54d10be8eb56f67b3a8ce P 6ad404efe189443cb1058eaf35888347 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:07.481173 16762 raft_consensus.cc:2258] T c89a88f56fb54d10be8eb56f67b3a8ce P 6ad404efe189443cb1058eaf35888347 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:07.497260 16762 tablet_server.cc:194] TabletServer@127.16.94.129:0 shutdown complete.
I20240503 14:12:07.502076 16762 tablet_server.cc:178] TabletServer@127.16.94.130:0 shutting down...
W20240503 14:12:07.506268 17377 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240503 14:12:07.506389 17377 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240503 14:12:07.513165 16762 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 14:12:07.513476 16762 tablet_replica.cc:330] T f085937f65ad428aa11585f416dd9c54 P 09f6d631be3e444c96284a839becc783: stopping tablet replica
I20240503 14:12:07.514010 16762 raft_consensus.cc:2229] T f085937f65ad428aa11585f416dd9c54 P 09f6d631be3e444c96284a839becc783 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:07.514351 16762 raft_consensus.cc:2258] T f085937f65ad428aa11585f416dd9c54 P 09f6d631be3e444c96284a839becc783 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:07.514997 16762 tablet_replica.cc:330] T c89a88f56fb54d10be8eb56f67b3a8ce P 09f6d631be3e444c96284a839becc783: stopping tablet replica
I20240503 14:12:07.515187 16762 raft_consensus.cc:2229] T c89a88f56fb54d10be8eb56f67b3a8ce P 09f6d631be3e444c96284a839becc783 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:07.515590 16762 raft_consensus.cc:2258] T c89a88f56fb54d10be8eb56f67b3a8ce P 09f6d631be3e444c96284a839becc783 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:07.516063 16762 tablet_replica.cc:330] T 9ca0f24b82244f7ebbcb264fd54545bd P 09f6d631be3e444c96284a839becc783: stopping tablet replica
I20240503 14:12:07.516234 16762 raft_consensus.cc:2229] T 9ca0f24b82244f7ebbcb264fd54545bd P 09f6d631be3e444c96284a839becc783 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:07.516376 16762 raft_consensus.cc:2258] T 9ca0f24b82244f7ebbcb264fd54545bd P 09f6d631be3e444c96284a839becc783 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:07.516865 16762 tablet_replica.cc:330] T 662fbeaef8a64215a1e17b6993c5189b P 09f6d631be3e444c96284a839becc783: stopping tablet replica
I20240503 14:12:07.517012 16762 raft_consensus.cc:2229] T 662fbeaef8a64215a1e17b6993c5189b P 09f6d631be3e444c96284a839becc783 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:07.517170 16762 raft_consensus.cc:2258] T 662fbeaef8a64215a1e17b6993c5189b P 09f6d631be3e444c96284a839becc783 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:07.517647 16762 tablet_replica.cc:330] T 121af11b035d49dd8a2f12cc8eb7c583 P 09f6d631be3e444c96284a839becc783: stopping tablet replica
I20240503 14:12:07.517843 16762 raft_consensus.cc:2229] T 121af11b035d49dd8a2f12cc8eb7c583 P 09f6d631be3e444c96284a839becc783 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:07.518040 16762 raft_consensus.cc:2258] T 121af11b035d49dd8a2f12cc8eb7c583 P 09f6d631be3e444c96284a839becc783 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:07.518672 16762 tablet_replica.cc:330] T ce7190aa241648008185b18f4a5b881f P 09f6d631be3e444c96284a839becc783: stopping tablet replica
I20240503 14:12:07.518811 16762 raft_consensus.cc:2229] T ce7190aa241648008185b18f4a5b881f P 09f6d631be3e444c96284a839becc783 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:07.518983 16762 raft_consensus.cc:2258] T ce7190aa241648008185b18f4a5b881f P 09f6d631be3e444c96284a839becc783 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:07.519623 16762 tablet_replica.cc:330] T 4b64a9a4099e47d98fd1534439b1c80c P 09f6d631be3e444c96284a839becc783: stopping tablet replica
I20240503 14:12:07.519788 16762 raft_consensus.cc:2229] T 4b64a9a4099e47d98fd1534439b1c80c P 09f6d631be3e444c96284a839becc783 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:07.520038 16762 raft_consensus.cc:2258] T 4b64a9a4099e47d98fd1534439b1c80c P 09f6d631be3e444c96284a839becc783 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:07.520473 16762 tablet_replica.cc:330] T 059a1f25723d4ba0be4cbb7185517bed P 09f6d631be3e444c96284a839becc783: stopping tablet replica
I20240503 14:12:07.520623 16762 raft_consensus.cc:2229] T 059a1f25723d4ba0be4cbb7185517bed P 09f6d631be3e444c96284a839becc783 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:07.520820 16762 raft_consensus.cc:2258] T 059a1f25723d4ba0be4cbb7185517bed P 09f6d631be3e444c96284a839becc783 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:07.527159 16762 tablet_server.cc:194] TabletServer@127.16.94.130:0 shutdown complete.
I20240503 14:12:07.532346 16762 tablet_server.cc:178] TabletServer@127.16.94.131:0 shutting down...
W20240503 14:12:07.534835 17377 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240503 14:12:07.535177 17377 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240503 14:12:07.541924 16762 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 14:12:07.542165 16762 tablet_replica.cc:330] T 059a1f25723d4ba0be4cbb7185517bed P 99bfdcfb9ee349ea9eb57b80fc86c71b: stopping tablet replica
I20240503 14:12:07.542348 16762 raft_consensus.cc:2229] T 059a1f25723d4ba0be4cbb7185517bed P 99bfdcfb9ee349ea9eb57b80fc86c71b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:07.542537 16762 raft_consensus.cc:2258] T 059a1f25723d4ba0be4cbb7185517bed P 99bfdcfb9ee349ea9eb57b80fc86c71b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:07.543264 16762 tablet_replica.cc:330] T f085937f65ad428aa11585f416dd9c54 P 99bfdcfb9ee349ea9eb57b80fc86c71b: stopping tablet replica
I20240503 14:12:07.543406 16762 raft_consensus.cc:2229] T f085937f65ad428aa11585f416dd9c54 P 99bfdcfb9ee349ea9eb57b80fc86c71b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:07.543522 16762 raft_consensus.cc:2258] T f085937f65ad428aa11585f416dd9c54 P 99bfdcfb9ee349ea9eb57b80fc86c71b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:07.543937 16762 tablet_replica.cc:330] T ce7190aa241648008185b18f4a5b881f P 99bfdcfb9ee349ea9eb57b80fc86c71b: stopping tablet replica
I20240503 14:12:07.544041 16762 raft_consensus.cc:2229] T ce7190aa241648008185b18f4a5b881f P 99bfdcfb9ee349ea9eb57b80fc86c71b [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:07.544348 16762 raft_consensus.cc:2258] T ce7190aa241648008185b18f4a5b881f P 99bfdcfb9ee349ea9eb57b80fc86c71b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:07.544845 16762 tablet_replica.cc:330] T 4b64a9a4099e47d98fd1534439b1c80c P 99bfdcfb9ee349ea9eb57b80fc86c71b: stopping tablet replica
I20240503 14:12:07.544965 16762 raft_consensus.cc:2229] T 4b64a9a4099e47d98fd1534439b1c80c P 99bfdcfb9ee349ea9eb57b80fc86c71b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:07.545075 16762 raft_consensus.cc:2258] T 4b64a9a4099e47d98fd1534439b1c80c P 99bfdcfb9ee349ea9eb57b80fc86c71b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:07.545424 16762 tablet_replica.cc:330] T c89a88f56fb54d10be8eb56f67b3a8ce P 99bfdcfb9ee349ea9eb57b80fc86c71b: stopping tablet replica
I20240503 14:12:07.545527 16762 raft_consensus.cc:2229] T c89a88f56fb54d10be8eb56f67b3a8ce P 99bfdcfb9ee349ea9eb57b80fc86c71b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:07.545724 16762 raft_consensus.cc:2258] T c89a88f56fb54d10be8eb56f67b3a8ce P 99bfdcfb9ee349ea9eb57b80fc86c71b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:07.546135 16762 tablet_replica.cc:330] T 9ca0f24b82244f7ebbcb264fd54545bd P 99bfdcfb9ee349ea9eb57b80fc86c71b: stopping tablet replica
I20240503 14:12:07.546247 16762 raft_consensus.cc:2229] T 9ca0f24b82244f7ebbcb264fd54545bd P 99bfdcfb9ee349ea9eb57b80fc86c71b [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:07.546437 16762 raft_consensus.cc:2258] T 9ca0f24b82244f7ebbcb264fd54545bd P 99bfdcfb9ee349ea9eb57b80fc86c71b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:07.546800 16762 tablet_replica.cc:330] T 121af11b035d49dd8a2f12cc8eb7c583 P 99bfdcfb9ee349ea9eb57b80fc86c71b: stopping tablet replica
I20240503 14:12:07.546916 16762 raft_consensus.cc:2229] T 121af11b035d49dd8a2f12cc8eb7c583 P 99bfdcfb9ee349ea9eb57b80fc86c71b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:07.547051 16762 raft_consensus.cc:2258] T 121af11b035d49dd8a2f12cc8eb7c583 P 99bfdcfb9ee349ea9eb57b80fc86c71b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:07.547616 16762 tablet_replica.cc:330] T 662fbeaef8a64215a1e17b6993c5189b P 99bfdcfb9ee349ea9eb57b80fc86c71b: stopping tablet replica
I20240503 14:12:07.547726 16762 raft_consensus.cc:2229] T 662fbeaef8a64215a1e17b6993c5189b P 99bfdcfb9ee349ea9eb57b80fc86c71b [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:07.547930 16762 raft_consensus.cc:2258] T 662fbeaef8a64215a1e17b6993c5189b P 99bfdcfb9ee349ea9eb57b80fc86c71b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:07.563356 16762 tablet_server.cc:194] TabletServer@127.16.94.131:0 shutdown complete.
I20240503 14:12:07.567196 16762 tablet_server.cc:178] TabletServer@127.16.94.132:0 shutting down...
I20240503 14:12:07.575479 16762 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 14:12:07.575829 16762 tablet_replica.cc:330] T 121af11b035d49dd8a2f12cc8eb7c583 P f03dcea3c66540f98f36ae3540b4d603: stopping tablet replica
I20240503 14:12:07.576027 16762 raft_consensus.cc:2229] T 121af11b035d49dd8a2f12cc8eb7c583 P f03dcea3c66540f98f36ae3540b4d603 [term 1 LEARNER]: Raft consensus shutting down.
I20240503 14:12:07.576189 16762 raft_consensus.cc:2258] T 121af11b035d49dd8a2f12cc8eb7c583 P f03dcea3c66540f98f36ae3540b4d603 [term 1 LEARNER]: Raft consensus is shut down!
I20240503 14:12:07.576825 16762 tablet_replica.cc:330] T ce7190aa241648008185b18f4a5b881f P f03dcea3c66540f98f36ae3540b4d603: stopping tablet replica
I20240503 14:12:07.576968 16762 raft_consensus.cc:2229] T ce7190aa241648008185b18f4a5b881f P f03dcea3c66540f98f36ae3540b4d603 [term 1 LEARNER]: Raft consensus shutting down.
I20240503 14:12:07.577117 16762 raft_consensus.cc:2258] T ce7190aa241648008185b18f4a5b881f P f03dcea3c66540f98f36ae3540b4d603 [term 1 LEARNER]: Raft consensus is shut down!
I20240503 14:12:07.577600 16762 tablet_replica.cc:330] T c89a88f56fb54d10be8eb56f67b3a8ce P f03dcea3c66540f98f36ae3540b4d603: stopping tablet replica
I20240503 14:12:07.577764 16762 raft_consensus.cc:2229] T c89a88f56fb54d10be8eb56f67b3a8ce P f03dcea3c66540f98f36ae3540b4d603 [term 1 LEARNER]: Raft consensus shutting down.
I20240503 14:12:07.577888 16762 raft_consensus.cc:2258] T c89a88f56fb54d10be8eb56f67b3a8ce P f03dcea3c66540f98f36ae3540b4d603 [term 1 LEARNER]: Raft consensus is shut down!
I20240503 14:12:07.578282 16762 tablet_replica.cc:330] T 059a1f25723d4ba0be4cbb7185517bed P f03dcea3c66540f98f36ae3540b4d603: stopping tablet replica
I20240503 14:12:07.578405 16762 raft_consensus.cc:2229] T 059a1f25723d4ba0be4cbb7185517bed P f03dcea3c66540f98f36ae3540b4d603 [term 1 LEARNER]: Raft consensus shutting down.
I20240503 14:12:07.578536 16762 raft_consensus.cc:2258] T 059a1f25723d4ba0be4cbb7185517bed P f03dcea3c66540f98f36ae3540b4d603 [term 1 LEARNER]: Raft consensus is shut down!
I20240503 14:12:07.592231 16762 tablet_server.cc:194] TabletServer@127.16.94.132:0 shutdown complete.
I20240503 14:12:07.595957 16762 master.cc:537] Master@127.16.94.190:44937 shutting down...
I20240503 14:12:07.603525 16762 raft_consensus.cc:2229] T 00000000000000000000000000000000 P ed158d547a714aa1be8b2e669157b58e [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:07.603729 16762 raft_consensus.cc:2258] T 00000000000000000000000000000000 P ed158d547a714aa1be8b2e669157b58e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:07.603817 16762 tablet_replica.cc:330] T 00000000000000000000000000000000 P ed158d547a714aa1be8b2e669157b58e: stopping tablet replica
I20240503 14:12:07.617796 16762 master.cc:552] Master@127.16.94.190:44937 shutdown complete.
[ OK ] AutoRebalancerTest.AutoRebalancingTurnOffAndOn (9346 ms)
[ RUN ] AutoRebalancerTest.NextLeaderResumesAutoRebalancing
I20240503 14:12:07.682631 16762 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.16.94.190:43583,127.16.94.189:35539,127.16.94.188:39819
I20240503 14:12:07.683247 16762 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:07.687001 17834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:07.687536 17833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:07.688304 17836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:07.689024 16762 server_base.cc:1034] running on GCE node
I20240503 14:12:07.689324 16762 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:12:07.689388 16762 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:12:07.689458 16762 hybrid_clock.cc:648] HybridClock initialized: now 1714745527689426 us; error 0 us; skew 500 ppm
I20240503 14:12:07.689610 16762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:07.691051 16762 webserver.cc:460] Webserver started at http://127.16.94.190:40459/ using document root <none> and password file <none>
I20240503 14:12:07.691293 16762 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:07.691390 16762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:07.691524 16762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:07.692214 16762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714745515241994-16762-0/minicluster-data/master-0-root/instance:
uuid: "7a6eda5ef7244439b207e95121ecaa54"
format_stamp: "Formatted at 2024-05-03 14:12:07 on dist-test-slave-v27g"
I20240503 14:12:07.694873 16762 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20240503 14:12:07.696483 17841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:07.696915 16762 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240503 14:12:07.697014 16762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714745515241994-16762-0/minicluster-data/master-0-root
uuid: "7a6eda5ef7244439b207e95121ecaa54"
format_stamp: "Formatted at 2024-05-03 14:12:07 on dist-test-slave-v27g"
I20240503 14:12:07.697129 16762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714745515241994-16762-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714745515241994-16762-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714745515241994-16762-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:07.719823 16762 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:07.720404 16762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:07.733232 16762 rpc_server.cc:307] RPC server started. Bound to: 127.16.94.190:43583
I20240503 14:12:07.733265 17892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.94.190:43583 every 8 connection(s)
I20240503 14:12:07.734611 16762 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 14:12:07.734676 17893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:07.738073 17893 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.94.190" port: 43583 } has no permanent_uuid. Determining permanent_uuid...
W20240503 14:12:07.738150 17896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:07.738590 17895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:07.739398 17898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:07.740187 16762 server_base.cc:1034] running on GCE node
I20240503 14:12:07.740453 16762 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:12:07.740546 16762 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:12:07.740623 16762 hybrid_clock.cc:648] HybridClock initialized: now 1714745527740585 us; error 0 us; skew 500 ppm
I20240503 14:12:07.740828 16762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:07.742370 16762 webserver.cc:460] Webserver started at http://127.16.94.189:34863/ using document root <none> and password file <none>
I20240503 14:12:07.742612 16762 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:07.742759 16762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:07.742914 16762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:07.743693 16762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714745515241994-16762-0/minicluster-data/master-1-root/instance:
uuid: "54e8c1e857c0443caea6bc05c1ce71e6"
format_stamp: "Formatted at 2024-05-03 14:12:07 on dist-test-slave-v27g"
I20240503 14:12:07.746464 17893 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.94.189" port: 35539 } has no permanent_uuid. Determining permanent_uuid...
I20240503 14:12:07.746878 16762 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
W20240503 14:12:07.748481 17893 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.16.94.189:35539: Network error: Client connection negotiation failed: client connection to 127.16.94.189:35539: connect: Connection refused (error 111)
I20240503 14:12:07.748975 17906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:07.749425 16762 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240503 14:12:07.749557 16762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714745515241994-16762-0/minicluster-data/master-1-root
uuid: "54e8c1e857c0443caea6bc05c1ce71e6"
format_stamp: "Formatted at 2024-05-03 14:12:07 on dist-test-slave-v27g"
I20240503 14:12:07.749729 16762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714745515241994-16762-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714745515241994-16762-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714745515241994-16762-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:07.781826 16762 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:07.782426 16762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:07.794718 17893 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.94.189" port: 35539 } attempt: 1
I20240503 14:12:07.796321 16762 rpc_server.cc:307] RPC server started. Bound to: 127.16.94.189:35539
I20240503 14:12:07.796370 17957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.94.189:35539 every 8 connection(s)
I20240503 14:12:07.798350 16762 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 14:12:07.798417 17959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:12:07.802714 17961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:07.804253 17893 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.94.188" port: 39819 } has no permanent_uuid. Determining permanent_uuid...
I20240503 14:12:07.804265 17959 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.94.190" port: 43583 } has no permanent_uuid. Determining permanent_uuid...
W20240503 14:12:07.804975 17962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:07.806090 17964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:07.807603 17893 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.16.94.188:39819: Network error: Client connection negotiation failed: client connection to 127.16.94.188:39819: connect: Connection refused (error 111)
I20240503 14:12:07.809530 16762 server_base.cc:1034] running on GCE node
I20240503 14:12:07.809931 16762 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:12:07.810029 16762 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:12:07.810112 16762 hybrid_clock.cc:648] HybridClock initialized: now 1714745527810070 us; error 0 us; skew 500 ppm
I20240503 14:12:07.810310 16762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:07.811749 17959 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.94.189" port: 35539 } has no permanent_uuid. Determining permanent_uuid...
I20240503 14:12:07.811836 16762 webserver.cc:460] Webserver started at http://127.16.94.188:44723/ using document root <none> and password file <none>
I20240503 14:12:07.812068 16762 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:07.812177 16762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:07.812325 16762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:07.813149 16762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714745515241994-16762-0/minicluster-data/master-2-root/instance:
uuid: "61252778e375497b838aed6a503c3559"
format_stamp: "Formatted at 2024-05-03 14:12:07 on dist-test-slave-v27g"
I20240503 14:12:07.816404 16762 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20240503 14:12:07.817258 17959 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.94.188" port: 39819 } has no permanent_uuid. Determining permanent_uuid...
I20240503 14:12:07.818416 17970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:07.818799 16762 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240503 14:12:07.818913 16762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714745515241994-16762-0/minicluster-data/master-2-root
uuid: "61252778e375497b838aed6a503c3559"
format_stamp: "Formatted at 2024-05-03 14:12:07 on dist-test-slave-v27g"
I20240503 14:12:07.819078 16762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714745515241994-16762-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714745515241994-16762-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714745515241994-16762-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20240503 14:12:07.819178 17959 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.16.94.188:39819: Network error: Client connection negotiation failed: client connection to 127.16.94.188:39819: connect: Connection refused (error 111)
I20240503 14:12:07.829828 17893 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.94.188" port: 39819 } attempt: 1
W20240503 14:12:07.831378 17893 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.16.94.188:39819: Network error: Client connection negotiation failed: client connection to 127.16.94.188:39819: connect: Connection refused (error 111)
I20240503 14:12:07.844724 16762 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:07.845315 16762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:07.851361 17959 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.94.188" port: 39819 } attempt: 1
W20240503 14:12:07.853367 17959 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.16.94.188:39819: Network error: Client connection negotiation failed: client connection to 127.16.94.188:39819: connect: Connection refused (error 111)
I20240503 14:12:07.857338 16762 rpc_server.cc:307] RPC server started. Bound to: 127.16.94.188:39819
I20240503 14:12:07.857370 18023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.94.188:39819 every 8 connection(s)
I20240503 14:12:07.858273 16762 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 14:12:07.858598 18024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:07.860746 18024 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.94.190" port: 43583 } has no permanent_uuid. Determining permanent_uuid...
I20240503 14:12:07.866277 18024 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.94.189" port: 35539 } has no permanent_uuid. Determining permanent_uuid...
I20240503 14:12:07.870899 18024 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.94.188" port: 39819 } has no permanent_uuid. Determining permanent_uuid...
I20240503 14:12:07.879562 18024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61252778e375497b838aed6a503c3559: Bootstrap starting.
I20240503 14:12:07.881385 18024 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 61252778e375497b838aed6a503c3559: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:07.882943 18024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61252778e375497b838aed6a503c3559: No bootstrap required, opened a new log
I20240503 14:12:07.883476 18024 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61252778e375497b838aed6a503c3559 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6eda5ef7244439b207e95121ecaa54" member_type: VOTER last_known_addr { host: "127.16.94.190" port: 43583 } } peers { permanent_uuid: "54e8c1e857c0443caea6bc05c1ce71e6" member_type: VOTER last_known_addr { host: "127.16.94.189" port: 35539 } } peers { permanent_uuid: "61252778e375497b838aed6a503c3559" member_type: VOTER last_known_addr { host: "127.16.94.188" port: 39819 } }
I20240503 14:12:07.883637 18024 raft_consensus.cc:374] T 00000000000000000000000000000000 P 61252778e375497b838aed6a503c3559 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:07.883690 18024 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61252778e375497b838aed6a503c3559 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61252778e375497b838aed6a503c3559, State: Initialized, Role: FOLLOWER
I20240503 14:12:07.883874 18024 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61252778e375497b838aed6a503c3559 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6eda5ef7244439b207e95121ecaa54" member_type: VOTER last_known_addr { host: "127.16.94.190" port: 43583 } } peers { permanent_uuid: "54e8c1e857c0443caea6bc05c1ce71e6" member_type: VOTER last_known_addr { host: "127.16.94.189" port: 35539 } } peers { permanent_uuid: "61252778e375497b838aed6a503c3559" member_type: VOTER last_known_addr { host: "127.16.94.188" port: 39819 } }
I20240503 14:12:07.884414 18029 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61252778e375497b838aed6a503c3559 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6eda5ef7244439b207e95121ecaa54" member_type: VOTER last_known_addr { host: "127.16.94.190" port: 43583 } } peers { permanent_uuid: "54e8c1e857c0443caea6bc05c1ce71e6" member_type: VOTER last_known_addr { host: "127.16.94.189" port: 35539 } } peers { permanent_uuid: "61252778e375497b838aed6a503c3559" member_type: VOTER last_known_addr { host: "127.16.94.188" port: 39819 } } }
I20240503 14:12:07.884601 18024 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61252778e375497b838aed6a503c3559 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 14:12:07.884608 18029 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61252778e375497b838aed6a503c3559 [sys.catalog]: This master's current role is: FOLLOWER
W20240503 14:12:07.888716 18040 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 61252778e375497b838aed6a503c3559: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 14:12:07.888808 18040 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240503 14:12:07.904604 17959 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.94.188" port: 39819 } attempt: 2
I20240503 14:12:07.910575 17893 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.94.188" port: 39819 } attempt: 2
I20240503 14:12:07.912534 17959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54e8c1e857c0443caea6bc05c1ce71e6: Bootstrap starting.
I20240503 14:12:07.914121 17959 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54e8c1e857c0443caea6bc05c1ce71e6: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:07.916131 17959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54e8c1e857c0443caea6bc05c1ce71e6: No bootstrap required, opened a new log
I20240503 14:12:07.916965 17959 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54e8c1e857c0443caea6bc05c1ce71e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6eda5ef7244439b207e95121ecaa54" member_type: VOTER last_known_addr { host: "127.16.94.190" port: 43583 } } peers { permanent_uuid: "54e8c1e857c0443caea6bc05c1ce71e6" member_type: VOTER last_known_addr { host: "127.16.94.189" port: 35539 } } peers { permanent_uuid: "61252778e375497b838aed6a503c3559" member_type: VOTER last_known_addr { host: "127.16.94.188" port: 39819 } }
I20240503 14:12:07.917205 17959 raft_consensus.cc:374] T 00000000000000000000000000000000 P 54e8c1e857c0443caea6bc05c1ce71e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:07.917289 17959 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54e8c1e857c0443caea6bc05c1ce71e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54e8c1e857c0443caea6bc05c1ce71e6, State: Initialized, Role: FOLLOWER
I20240503 14:12:07.917532 17959 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54e8c1e857c0443caea6bc05c1ce71e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6eda5ef7244439b207e95121ecaa54" member_type: VOTER last_known_addr { host: "127.16.94.190" port: 43583 } } peers { permanent_uuid: "54e8c1e857c0443caea6bc05c1ce71e6" member_type: VOTER last_known_addr { host: "127.16.94.189" port: 35539 } } peers { permanent_uuid: "61252778e375497b838aed6a503c3559" member_type: VOTER last_known_addr { host: "127.16.94.188" port: 39819 } }
I20240503 14:12:07.918268 18042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54e8c1e857c0443caea6bc05c1ce71e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6eda5ef7244439b207e95121ecaa54" member_type: VOTER last_known_addr { host: "127.16.94.190" port: 43583 } } peers { permanent_uuid: "54e8c1e857c0443caea6bc05c1ce71e6" member_type: VOTER last_known_addr { host: "127.16.94.189" port: 35539 } } peers { permanent_uuid: "61252778e375497b838aed6a503c3559" member_type: VOTER last_known_addr { host: "127.16.94.188" port: 39819 } } }
I20240503 14:12:07.918473 18042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54e8c1e857c0443caea6bc05c1ce71e6 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 14:12:07.918570 17959 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54e8c1e857c0443caea6bc05c1ce71e6 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 14:12:07.919502 17893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a6eda5ef7244439b207e95121ecaa54: Bootstrap starting.
I20240503 14:12:07.921465 17893 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a6eda5ef7244439b207e95121ecaa54: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:07.923733 17893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a6eda5ef7244439b207e95121ecaa54: No bootstrap required, opened a new log
I20240503 14:12:07.924335 17893 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a6eda5ef7244439b207e95121ecaa54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6eda5ef7244439b207e95121ecaa54" member_type: VOTER last_known_addr { host: "127.16.94.190" port: 43583 } } peers { permanent_uuid: "54e8c1e857c0443caea6bc05c1ce71e6" member_type: VOTER last_known_addr { host: "127.16.94.189" port: 35539 } } peers { permanent_uuid: "61252778e375497b838aed6a503c3559" member_type: VOTER last_known_addr { host: "127.16.94.188" port: 39819 } }
W20240503 14:12:07.924440 18054 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 54e8c1e857c0443caea6bc05c1ce71e6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20240503 14:12:07.924499 17893 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a6eda5ef7244439b207e95121ecaa54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240503 14:12:07.924571 18054 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240503 14:12:07.924574 17893 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a6eda5ef7244439b207e95121ecaa54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a6eda5ef7244439b207e95121ecaa54, State: Initialized, Role: FOLLOWER
I20240503 14:12:07.924796 17893 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a6eda5ef7244439b207e95121ecaa54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6eda5ef7244439b207e95121ecaa54" member_type: VOTER last_known_addr { host: "127.16.94.190" port: 43583 } } peers { permanent_uuid: "54e8c1e857c0443caea6bc05c1ce71e6" member_type: VOTER last_known_addr { host: "127.16.94.189" port: 35539 } } peers { permanent_uuid: "61252778e375497b838aed6a503c3559" member_type: VOTER last_known_addr { host: "127.16.94.188" port: 39819 } }
I20240503 14:12:07.925350 18055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a6eda5ef7244439b207e95121ecaa54 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6eda5ef7244439b207e95121ecaa54" member_type: VOTER last_known_addr { host: "127.16.94.190" port: 43583 } } peers { permanent_uuid: "54e8c1e857c0443caea6bc05c1ce71e6" member_type: VOTER last_known_addr { host: "127.16.94.189" port: 35539 } } peers { permanent_uuid: "61252778e375497b838aed6a503c3559" member_type: VOTER last_known_addr { host: "127.16.94.188" port: 39819 } } }
I20240503 14:12:07.925539 17893 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a6eda5ef7244439b207e95121ecaa54 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 14:12:07.925556 18055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a6eda5ef7244439b207e95121ecaa54 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 14:12:07.928908 16762 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
I20240503 14:12:07.929031 16762 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
I20240503 14:12:07.929996 16762 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:07.930279 18066 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7a6eda5ef7244439b207e95121ecaa54: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 14:12:07.930378 18066 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20240503 14:12:07.933512 18067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:07.934337 18070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:07.935074 18068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:07.935828 16762 server_base.cc:1034] running on GCE node
I20240503 14:12:07.936110 16762 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:12:07.936195 16762 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:12:07.936257 16762 hybrid_clock.cc:648] HybridClock initialized: now 1714745527936239 us; error 0 us; skew 500 ppm
I20240503 14:12:07.936417 16762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:07.937724 16762 webserver.cc:460] Webserver started at http://127.16.94.129:44779/ using document root <none> and password file <none>
I20240503 14:12:07.937955 16762 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:07.938036 16762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:07.938197 16762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:07.938864 16762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714745515241994-16762-0/minicluster-data/ts-0-root/instance:
uuid: "0c3fed516d2349dbaef0ef9e25690283"
format_stamp: "Formatted at 2024-05-03 14:12:07 on dist-test-slave-v27g"
I20240503 14:12:07.941287 16762 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 14:12:07.943001 18075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:07.943336 16762 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240503 14:12:07.943446 16762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714745515241994-16762-0/minicluster-data/ts-0-root
uuid: "0c3fed516d2349dbaef0ef9e25690283"
format_stamp: "Formatted at 2024-05-03 14:12:07 on dist-test-slave-v27g"
I20240503 14:12:07.943564 16762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714745515241994-16762-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714745515241994-16762-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714745515241994-16762-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:07.948643 16762 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:07.949146 16762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:07.949769 16762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:07.950783 16762 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:07.950872 16762 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:07.950974 16762 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:07.951052 16762 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:07.965256 16762 rpc_server.cc:307] RPC server started. Bound to: 127.16.94.129:46657
I20240503 14:12:07.965277 18138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.94.129:46657 every 8 connection(s)
I20240503 14:12:07.974331 16762 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 14:12:07.977319 18139 heartbeater.cc:347] Connected to a master server at 127.16.94.188:39819
I20240503 14:12:07.977499 18139 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:07.978165 18139 heartbeater.cc:511] Master 127.16.94.188:39819 requested a full tablet report, sending...
I20240503 14:12:07.979583 17988 ts_manager.cc:194] Registered new tserver with Master: 0c3fed516d2349dbaef0ef9e25690283 (127.16.94.129:46657)
W20240503 14:12:07.987146 18146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:07.987838 18141 heartbeater.cc:347] Connected to a master server at 127.16.94.189:35539
I20240503 14:12:07.988068 18141 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:07.988776 18141 heartbeater.cc:511] Master 127.16.94.189:35539 requested a full tablet report, sending...
I20240503 14:12:07.989184 18140 heartbeater.cc:347] Connected to a master server at 127.16.94.190:43583
I20240503 14:12:07.989372 18140 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:07.989953 18140 heartbeater.cc:511] Master 127.16.94.190:43583 requested a full tablet report, sending...
W20240503 14:12:07.990075 18147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:07.990329 17923 ts_manager.cc:194] Registered new tserver with Master: 0c3fed516d2349dbaef0ef9e25690283 (127.16.94.129:46657)
I20240503 14:12:07.990577 16762 server_base.cc:1034] running on GCE node
I20240503 14:12:07.991060 17858 ts_manager.cc:194] Registered new tserver with Master: 0c3fed516d2349dbaef0ef9e25690283 (127.16.94.129:46657)
W20240503 14:12:07.991503 18149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:07.991858 16762 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:12:07.991936 16762 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:12:07.992017 16762 hybrid_clock.cc:648] HybridClock initialized: now 1714745527991981 us; error 0 us; skew 500 ppm
I20240503 14:12:07.992161 16762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:07.993512 16762 webserver.cc:460] Webserver started at http://127.16.94.130:45721/ using document root <none> and password file <none>
I20240503 14:12:07.993762 16762 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:07.993860 16762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:07.993999 16762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:07.994628 16762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714745515241994-16762-0/minicluster-data/ts-1-root/instance:
uuid: "d90f81c6cc4049df9f4ea38dc5fd6308"
format_stamp: "Formatted at 2024-05-03 14:12:07 on dist-test-slave-v27g"
I20240503 14:12:07.996917 16762 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 14:12:07.998597 18154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:07.998912 16762 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 14:12:07.999049 16762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714745515241994-16762-0/minicluster-data/ts-1-root
uuid: "d90f81c6cc4049df9f4ea38dc5fd6308"
format_stamp: "Formatted at 2024-05-03 14:12:07 on dist-test-slave-v27g"
I20240503 14:12:07.999179 16762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714745515241994-16762-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714745515241994-16762-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714745515241994-16762-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:08.018321 16762 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:08.018952 16762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:08.019554 16762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:08.020570 16762 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:08.020639 16762 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:08.020761 16762 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:08.020817 16762 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:08.039723 16762 rpc_server.cc:307] RPC server started. Bound to: 127.16.94.130:38429
I20240503 14:12:08.039779 18216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.94.130:38429 every 8 connection(s)
I20240503 14:12:08.048528 16762 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 14:12:08.052106 18217 heartbeater.cc:347] Connected to a master server at 127.16.94.188:39819
I20240503 14:12:08.052302 18217 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:08.052963 18217 heartbeater.cc:511] Master 127.16.94.188:39819 requested a full tablet report, sending...
I20240503 14:12:08.054422 17988 ts_manager.cc:194] Registered new tserver with Master: d90f81c6cc4049df9f4ea38dc5fd6308 (127.16.94.130:38429)
W20240503 14:12:08.059258 18225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:08.060087 18224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:08.060626 18218 heartbeater.cc:347] Connected to a master server at 127.16.94.190:43583
I20240503 14:12:08.060796 18218 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:08.061339 18218 heartbeater.cc:511] Master 127.16.94.190:43583 requested a full tablet report, sending...
I20240503 14:12:08.062639 17858 ts_manager.cc:194] Registered new tserver with Master: d90f81c6cc4049df9f4ea38dc5fd6308 (127.16.94.130:38429)
I20240503 14:12:08.063604 18219 heartbeater.cc:347] Connected to a master server at 127.16.94.189:35539
I20240503 14:12:08.063719 18219 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:08.064116 18219 heartbeater.cc:511] Master 127.16.94.189:35539 requested a full tablet report, sending...
I20240503 14:12:08.065110 17923 ts_manager.cc:194] Registered new tserver with Master: d90f81c6cc4049df9f4ea38dc5fd6308 (127.16.94.130:38429)
W20240503 14:12:08.066069 18228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:08.066953 16762 server_base.cc:1034] running on GCE node
I20240503 14:12:08.067193 16762 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:12:08.067270 16762 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:12:08.067332 16762 hybrid_clock.cc:648] HybridClock initialized: now 1714745528067313 us; error 0 us; skew 500 ppm
I20240503 14:12:08.067504 16762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:08.068826 16762 webserver.cc:460] Webserver started at http://127.16.94.131:40751/ using document root <none> and password file <none>
I20240503 14:12:08.069097 16762 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:08.069180 16762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:08.069329 16762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:08.070174 16762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714745515241994-16762-0/minicluster-data/ts-2-root/instance:
uuid: "19ea341ea26043299e01c85bba6b77f1"
format_stamp: "Formatted at 2024-05-03 14:12:08 on dist-test-slave-v27g"
I20240503 14:12:08.072707 16762 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20240503 14:12:08.074321 18233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:08.074707 16762 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.002s
I20240503 14:12:08.074805 16762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714745515241994-16762-0/minicluster-data/ts-2-root
uuid: "19ea341ea26043299e01c85bba6b77f1"
format_stamp: "Formatted at 2024-05-03 14:12:08 on dist-test-slave-v27g"
I20240503 14:12:08.074899 16762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714745515241994-16762-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714745515241994-16762-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714745515241994-16762-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:08.092311 16762 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:08.092866 16762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:08.093474 16762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:08.094445 16762 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:08.094523 16762 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:08.094635 16762 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:08.094712 16762 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:08.112967 16762 rpc_server.cc:307] RPC server started. Bound to: 127.16.94.131:39743
I20240503 14:12:08.113014 18295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.94.131:39743 every 8 connection(s)
I20240503 14:12:08.127097 18297 heartbeater.cc:347] Connected to a master server at 127.16.94.190:43583
I20240503 14:12:08.127341 18297 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:08.127957 18297 heartbeater.cc:511] Master 127.16.94.190:43583 requested a full tablet report, sending...
I20240503 14:12:08.129501 17858 ts_manager.cc:194] Registered new tserver with Master: 19ea341ea26043299e01c85bba6b77f1 (127.16.94.131:39743)
I20240503 14:12:08.130414 18296 heartbeater.cc:347] Connected to a master server at 127.16.94.188:39819
I20240503 14:12:08.130524 18296 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:08.130962 18296 heartbeater.cc:511] Master 127.16.94.188:39819 requested a full tablet report, sending...
I20240503 14:12:08.132151 17988 ts_manager.cc:194] Registered new tserver with Master: 19ea341ea26043299e01c85bba6b77f1 (127.16.94.131:39743)
I20240503 14:12:08.132803 18298 heartbeater.cc:347] Connected to a master server at 127.16.94.189:35539
I20240503 14:12:08.132966 18298 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:08.133347 18298 heartbeater.cc:511] Master 127.16.94.189:35539 requested a full tablet report, sending...
I20240503 14:12:08.134182 17923 ts_manager.cc:194] Registered new tserver with Master: 19ea341ea26043299e01c85bba6b77f1 (127.16.94.131:39743)
I20240503 14:12:08.134678 16762 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.008034596s
I20240503 14:12:08.163084 18029 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61252778e375497b838aed6a503c3559 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:08.163300 18029 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61252778e375497b838aed6a503c3559 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6eda5ef7244439b207e95121ecaa54" member_type: VOTER last_known_addr { host: "127.16.94.190" port: 43583 } } peers { permanent_uuid: "54e8c1e857c0443caea6bc05c1ce71e6" member_type: VOTER last_known_addr { host: "127.16.94.189" port: 35539 } } peers { permanent_uuid: "61252778e375497b838aed6a503c3559" member_type: VOTER last_known_addr { host: "127.16.94.188" port: 39819 } }
I20240503 14:12:08.164113 18029 leader_election.cc:288] T 00000000000000000000000000000000 P 61252778e375497b838aed6a503c3559 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a6eda5ef7244439b207e95121ecaa54 (127.16.94.190:43583), 54e8c1e857c0443caea6bc05c1ce71e6 (127.16.94.189:35539)
I20240503 14:12:08.164645 17868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "61252778e375497b838aed6a503c3559" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a6eda5ef7244439b207e95121ecaa54" is_pre_election: true
I20240503 14:12:08.164698 17933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "61252778e375497b838aed6a503c3559" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54e8c1e857c0443caea6bc05c1ce71e6" is_pre_election: true
I20240503 14:12:08.164899 17868 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7a6eda5ef7244439b207e95121ecaa54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61252778e375497b838aed6a503c3559 in term 0.
I20240503 14:12:08.164984 17933 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 54e8c1e857c0443caea6bc05c1ce71e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61252778e375497b838aed6a503c3559 in term 0.
I20240503 14:12:08.165410 17974 leader_election.cc:302] T 00000000000000000000000000000000 P 61252778e375497b838aed6a503c3559 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 54e8c1e857c0443caea6bc05c1ce71e6, 61252778e375497b838aed6a503c3559, 7a6eda5ef7244439b207e95121ecaa54; no voters:
I20240503 14:12:08.165690 18029 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 61252778e375497b838aed6a503c3559 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:08.165807 18029 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61252778e375497b838aed6a503c3559 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:08.165903 18029 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61252778e375497b838aed6a503c3559 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:08.182936 18029 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61252778e375497b838aed6a503c3559 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6eda5ef7244439b207e95121ecaa54" member_type: VOTER last_known_addr { host: "127.16.94.190" port: 43583 } } peers { permanent_uuid: "54e8c1e857c0443caea6bc05c1ce71e6" member_type: VOTER last_known_addr { host: "127.16.94.189" port: 35539 } } peers { permanent_uuid: "61252778e375497b838aed6a503c3559" member_type: VOTER last_known_addr { host: "127.16.94.188" port: 39819 } }
I20240503 14:12:08.183650 18029 leader_election.cc:288] T 00000000000000000000000000000000 P 61252778e375497b838aed6a503c3559 [CANDIDATE]: Term 1 election: Requested vote from peers 7a6eda5ef7244439b207e95121ecaa54 (127.16.94.190:43583), 54e8c1e857c0443caea6bc05c1ce71e6 (127.16.94.189:35539)
I20240503 14:12:08.184227 17933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "61252778e375497b838aed6a503c3559" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54e8c1e857c0443caea6bc05c1ce71e6"
I20240503 14:12:08.184296 17868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "61252778e375497b838aed6a503c3559" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a6eda5ef7244439b207e95121ecaa54"
I20240503 14:12:08.184432 17933 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54e8c1e857c0443caea6bc05c1ce71e6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:08.184548 17868 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a6eda5ef7244439b207e95121ecaa54 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:08.287803 17933 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 54e8c1e857c0443caea6bc05c1ce71e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61252778e375497b838aed6a503c3559 in term 1.
I20240503 14:12:08.288360 17868 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7a6eda5ef7244439b207e95121ecaa54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61252778e375497b838aed6a503c3559 in term 1.
I20240503 14:12:08.288424 17974 leader_election.cc:302] T 00000000000000000000000000000000 P 61252778e375497b838aed6a503c3559 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54e8c1e857c0443caea6bc05c1ce71e6, 61252778e375497b838aed6a503c3559; no voters:
I20240503 14:12:08.288805 18029 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 61252778e375497b838aed6a503c3559 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:08.289438 18029 raft_consensus.cc:686] T 00000000000000000000000000000000 P 61252778e375497b838aed6a503c3559 [term 1 LEADER]: Becoming Leader. State: Replica: 61252778e375497b838aed6a503c3559, State: Running, Role: LEADER
I20240503 14:12:08.289746 18029 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61252778e375497b838aed6a503c3559 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6eda5ef7244439b207e95121ecaa54" member_type: VOTER last_known_addr { host: "127.16.94.190" port: 43583 } } peers { permanent_uuid: "54e8c1e857c0443caea6bc05c1ce71e6" member_type: VOTER last_known_addr { host: "127.16.94.189" port: 35539 } } peers { permanent_uuid: "61252778e375497b838aed6a503c3559" member_type: VOTER last_known_addr { host: "127.16.94.188" port: 39819 } }
I20240503 14:12:08.291241 18304 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61252778e375497b838aed6a503c3559 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61252778e375497b838aed6a503c3559. Latest consensus state: current_term: 1 leader_uuid: "61252778e375497b838aed6a503c3559" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6eda5ef7244439b207e95121ecaa54" member_type: VOTER last_known_addr { host: "127.16.94.190" port: 43583 } } peers { permanent_uuid: "54e8c1e857c0443caea6bc05c1ce71e6" member_type: VOTER last_known_addr { host: "127.16.94.189" port: 35539 } } peers { permanent_uuid: "61252778e375497b838aed6a503c3559" member_type: VOTER last_known_addr { host: "127.16.94.188" port: 39819 } } }
I20240503 14:12:08.291473 18304 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61252778e375497b838aed6a503c3559 [sys.catalog]: This master's current role is: LEADER
I20240503 14:12:08.292063 18306 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 14:12:08.293224 18306 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 14:12:08.295907 17868 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7a6eda5ef7244439b207e95121ecaa54 [term 1 FOLLOWER]: Refusing update from remote peer 61252778e375497b838aed6a503c3559: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:12:08.296002 17933 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 54e8c1e857c0443caea6bc05c1ce71e6 [term 1 FOLLOWER]: Refusing update from remote peer 61252778e375497b838aed6a503c3559: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:12:08.296414 18029 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 61252778e375497b838aed6a503c3559 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6eda5ef7244439b207e95121ecaa54" member_type: VOTER last_known_addr { host: "127.16.94.190" port: 43583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:08.296638 18304 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 61252778e375497b838aed6a503c3559 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54e8c1e857c0443caea6bc05c1ce71e6" member_type: VOTER last_known_addr { host: "127.16.94.189" port: 35539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:08.299635 18055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a6eda5ef7244439b207e95121ecaa54 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61252778e375497b838aed6a503c3559. Latest consensus state: current_term: 1 leader_uuid: "61252778e375497b838aed6a503c3559" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6eda5ef7244439b207e95121ecaa54" member_type: VOTER last_known_addr { host: "127.16.94.190" port: 43583 } } peers { permanent_uuid: "54e8c1e857c0443caea6bc05c1ce71e6" member_type: VOTER last_known_addr { host: "127.16.94.189" port: 35539 } } peers { permanent_uuid: "61252778e375497b838aed6a503c3559" member_type: VOTER last_known_addr { host: "127.16.94.188" port: 39819 } } }
I20240503 14:12:08.299850 18055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a6eda5ef7244439b207e95121ecaa54 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 14:12:08.301749 18029 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61252778e375497b838aed6a503c3559 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "61252778e375497b838aed6a503c3559" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6eda5ef7244439b207e95121ecaa54" member_type: VOTER last_known_addr { host: "127.16.94.190" port: 43583 } } peers { permanent_uuid: "54e8c1e857c0443caea6bc05c1ce71e6" member_type: VOTER last_known_addr { host: "127.16.94.189" port: 35539 } } peers { permanent_uuid: "61252778e375497b838aed6a503c3559" member_type: VOTER last_known_addr { host: "127.16.94.188" port: 39819 } } }
I20240503 14:12:08.302006 18029 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61252778e375497b838aed6a503c3559 [sys.catalog]: This master's current role is: LEADER
I20240503 14:12:08.302358 18306 catalog_manager.cc:1347] Generated new cluster ID: e03a07a9169d4531a47a50f9a0e1c699
I20240503 14:12:08.302453 18306 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 14:12:08.306291 18055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a6eda5ef7244439b207e95121ecaa54 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "61252778e375497b838aed6a503c3559" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6eda5ef7244439b207e95121ecaa54" member_type: VOTER last_known_addr { host: "127.16.94.190" port: 43583 } } peers { permanent_uuid: "54e8c1e857c0443caea6bc05c1ce71e6" member_type: VOTER last_known_addr { host: "127.16.94.189" port: 35539 } } peers { permanent_uuid: "61252778e375497b838aed6a503c3559" member_type: VOTER last_known_addr { host: "127.16.94.188" port: 39819 } } }
I20240503 14:12:08.306512 18055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a6eda5ef7244439b207e95121ecaa54 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 14:12:08.307267 18042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54e8c1e857c0443caea6bc05c1ce71e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61252778e375497b838aed6a503c3559. Latest consensus state: current_term: 1 leader_uuid: "61252778e375497b838aed6a503c3559" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6eda5ef7244439b207e95121ecaa54" member_type: VOTER last_known_addr { host: "127.16.94.190" port: 43583 } } peers { permanent_uuid: "54e8c1e857c0443caea6bc05c1ce71e6" member_type: VOTER last_known_addr { host: "127.16.94.189" port: 35539 } } peers { permanent_uuid: "61252778e375497b838aed6a503c3559" member_type: VOTER last_known_addr { host: "127.16.94.188" port: 39819 } } }
I20240503 14:12:08.307525 18042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54e8c1e857c0443caea6bc05c1ce71e6 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 14:12:08.308504 18029 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61252778e375497b838aed6a503c3559 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "61252778e375497b838aed6a503c3559" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6eda5ef7244439b207e95121ecaa54" member_type: VOTER last_known_addr { host: "127.16.94.190" port: 43583 } } peers { permanent_uuid: "54e8c1e857c0443caea6bc05c1ce71e6" member_type: VOTER last_known_addr { host: "127.16.94.189" port: 35539 } } peers { permanent_uuid: "61252778e375497b838aed6a503c3559" member_type: VOTER last_known_addr { host: "127.16.94.188" port: 39819 } } }
I20240503 14:12:08.308775 18029 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61252778e375497b838aed6a503c3559 [sys.catalog]: This master's current role is: LEADER
I20240503 14:12:08.309758 18042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54e8c1e857c0443caea6bc05c1ce71e6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "61252778e375497b838aed6a503c3559" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6eda5ef7244439b207e95121ecaa54" member_type: VOTER last_known_addr { host: "127.16.94.190" port: 43583 } } peers { permanent_uuid: "54e8c1e857c0443caea6bc05c1ce71e6" member_type: VOTER last_known_addr { host: "127.16.94.189" port: 35539 } } peers { permanent_uuid: "61252778e375497b838aed6a503c3559" member_type: VOTER last_known_addr { host: "127.16.94.188" port: 39819 } } }
I20240503 14:12:08.310022 18042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54e8c1e857c0443caea6bc05c1ce71e6 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 14:12:08.331192 18306 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 14:12:08.332003 18306 catalog_manager.cc:1504] Loading token signing keys...
I20240503 14:12:08.341378 18306 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 61252778e375497b838aed6a503c3559: Generated new TSK 0
I20240503 14:12:08.341671 18306 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 14:12:08.925938 18054 catalog_manager.cc:1259] Loaded cluster ID: e03a07a9169d4531a47a50f9a0e1c699
I20240503 14:12:08.926038 18054 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 54e8c1e857c0443caea6bc05c1ce71e6: loading cluster ID for follower catalog manager: success
I20240503 14:12:08.927960 18054 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 54e8c1e857c0443caea6bc05c1ce71e6: acquiring CA information for follower catalog manager: success
I20240503 14:12:08.929148 18054 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 54e8c1e857c0443caea6bc05c1ce71e6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240503 14:12:08.931308 18066 catalog_manager.cc:1259] Loaded cluster ID: e03a07a9169d4531a47a50f9a0e1c699
I20240503 14:12:08.931367 18066 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7a6eda5ef7244439b207e95121ecaa54: loading cluster ID for follower catalog manager: success
I20240503 14:12:08.933048 18066 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7a6eda5ef7244439b207e95121ecaa54: acquiring CA information for follower catalog manager: success
I20240503 14:12:08.933954 18066 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7a6eda5ef7244439b207e95121ecaa54: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240503 14:12:08.982452 17988 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53358
I20240503 14:12:09.057003 17988 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53360
I20240503 14:12:09.134585 17988 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53372
I20240503 14:12:09.887970 18039 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:12:09.937075 16762 test_util.cc:268] Using random seed: 1311845535
I20240503 14:12:09.953488 17988 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53378:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\252\252\252*\004\001\000TUUU""\004\001\000\252\252\252*\004\001\000TUUU"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:12:09.981189 18104 tablet_service.cc:1467] Processing CreateTablet for tablet e21fe982cace43169ce62a2ce91d88a5 (DEFAULT_TABLE table=test-workload [id=87bb46fdb45d4757ae04513926b680b5]), partition=RANGE (key) PARTITION VALUES < 715827882
I20240503 14:12:09.982086 18104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e21fe982cace43169ce62a2ce91d88a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:09.982321 18182 tablet_service.cc:1467] Processing CreateTablet for tablet b58941a606884f99886ac84b786b2add (DEFAULT_TABLE table=test-workload [id=87bb46fdb45d4757ae04513926b680b5]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20240503 14:12:09.982827 18182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b58941a606884f99886ac84b786b2add. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:09.983664 18139 heartbeater.cc:503] Master 127.16.94.188:39819 was elected leader, sending a full tablet report...
I20240503 14:12:09.988165 18261 tablet_service.cc:1467] Processing CreateTablet for tablet ffa7a5c99dd3474cac851118fe3db574 (DEFAULT_TABLE table=test-workload [id=87bb46fdb45d4757ae04513926b680b5]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20240503 14:12:09.988561 18261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffa7a5c99dd3474cac851118fe3db574. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:09.989948 18344 tablet_bootstrap.cc:492] T b58941a606884f99886ac84b786b2add P d90f81c6cc4049df9f4ea38dc5fd6308: Bootstrap starting.
I20240503 14:12:09.991916 18344 tablet_bootstrap.cc:654] T b58941a606884f99886ac84b786b2add P d90f81c6cc4049df9f4ea38dc5fd6308: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:09.992645 18345 tablet_bootstrap.cc:492] T e21fe982cace43169ce62a2ce91d88a5 P 0c3fed516d2349dbaef0ef9e25690283: Bootstrap starting.
I20240503 14:12:09.995200 18345 tablet_bootstrap.cc:654] T e21fe982cace43169ce62a2ce91d88a5 P 0c3fed516d2349dbaef0ef9e25690283: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:09.997205 18348 tablet_bootstrap.cc:492] T ffa7a5c99dd3474cac851118fe3db574 P 19ea341ea26043299e01c85bba6b77f1: Bootstrap starting.
I20240503 14:12:09.997617 18345 tablet_bootstrap.cc:492] T e21fe982cace43169ce62a2ce91d88a5 P 0c3fed516d2349dbaef0ef9e25690283: No bootstrap required, opened a new log
I20240503 14:12:09.997866 18345 ts_tablet_manager.cc:1367] T e21fe982cace43169ce62a2ce91d88a5 P 0c3fed516d2349dbaef0ef9e25690283: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 14:12:09.998508 18344 tablet_bootstrap.cc:492] T b58941a606884f99886ac84b786b2add P d90f81c6cc4049df9f4ea38dc5fd6308: No bootstrap required, opened a new log
I20240503 14:12:09.998661 18344 ts_tablet_manager.cc:1367] T b58941a606884f99886ac84b786b2add P d90f81c6cc4049df9f4ea38dc5fd6308: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20240503 14:12:09.998868 18345 raft_consensus.cc:348] T e21fe982cace43169ce62a2ce91d88a5 P 0c3fed516d2349dbaef0ef9e25690283 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c3fed516d2349dbaef0ef9e25690283" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 46657 } }
I20240503 14:12:09.998989 18348 tablet_bootstrap.cc:654] T ffa7a5c99dd3474cac851118fe3db574 P 19ea341ea26043299e01c85bba6b77f1: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:09.999082 18345 raft_consensus.cc:374] T e21fe982cace43169ce62a2ce91d88a5 P 0c3fed516d2349dbaef0ef9e25690283 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:09.999207 18345 raft_consensus.cc:729] T e21fe982cace43169ce62a2ce91d88a5 P 0c3fed516d2349dbaef0ef9e25690283 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c3fed516d2349dbaef0ef9e25690283, State: Initialized, Role: FOLLOWER
I20240503 14:12:09.999550 18344 raft_consensus.cc:348] T b58941a606884f99886ac84b786b2add P d90f81c6cc4049df9f4ea38dc5fd6308 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d90f81c6cc4049df9f4ea38dc5fd6308" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 38429 } }
I20240503 14:12:09.999552 18345 consensus_queue.cc:260] T e21fe982cace43169ce62a2ce91d88a5 P 0c3fed516d2349dbaef0ef9e25690283 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c3fed516d2349dbaef0ef9e25690283" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 46657 } }
I20240503 14:12:09.999745 18344 raft_consensus.cc:374] T b58941a606884f99886ac84b786b2add P d90f81c6cc4049df9f4ea38dc5fd6308 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:09.999863 18344 raft_consensus.cc:729] T b58941a606884f99886ac84b786b2add P d90f81c6cc4049df9f4ea38dc5fd6308 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d90f81c6cc4049df9f4ea38dc5fd6308, State: Initialized, Role: FOLLOWER
I20240503 14:12:10.000099 18344 consensus_queue.cc:260] T b58941a606884f99886ac84b786b2add P d90f81c6cc4049df9f4ea38dc5fd6308 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d90f81c6cc4049df9f4ea38dc5fd6308" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 38429 } }
I20240503 14:12:10.000283 18344 raft_consensus.cc:388] T b58941a606884f99886ac84b786b2add P d90f81c6cc4049df9f4ea38dc5fd6308 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:12:10.000339 18344 raft_consensus.cc:482] T b58941a606884f99886ac84b786b2add P d90f81c6cc4049df9f4ea38dc5fd6308 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:12:10.000414 18344 raft_consensus.cc:3037] T b58941a606884f99886ac84b786b2add P d90f81c6cc4049df9f4ea38dc5fd6308 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:10.000995 18348 tablet_bootstrap.cc:492] T ffa7a5c99dd3474cac851118fe3db574 P 19ea341ea26043299e01c85bba6b77f1: No bootstrap required, opened a new log
I20240503 14:12:10.001102 18348 ts_tablet_manager.cc:1367] T ffa7a5c99dd3474cac851118fe3db574 P 19ea341ea26043299e01c85bba6b77f1: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:12:10.001644 18348 raft_consensus.cc:348] T ffa7a5c99dd3474cac851118fe3db574 P 19ea341ea26043299e01c85bba6b77f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19ea341ea26043299e01c85bba6b77f1" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39743 } }
I20240503 14:12:10.001817 18348 raft_consensus.cc:374] T ffa7a5c99dd3474cac851118fe3db574 P 19ea341ea26043299e01c85bba6b77f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:10.001893 18348 raft_consensus.cc:729] T ffa7a5c99dd3474cac851118fe3db574 P 19ea341ea26043299e01c85bba6b77f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19ea341ea26043299e01c85bba6b77f1, State: Initialized, Role: FOLLOWER
I20240503 14:12:10.002090 18348 consensus_queue.cc:260] T ffa7a5c99dd3474cac851118fe3db574 P 19ea341ea26043299e01c85bba6b77f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19ea341ea26043299e01c85bba6b77f1" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39743 } }
I20240503 14:12:10.002250 18348 raft_consensus.cc:388] T ffa7a5c99dd3474cac851118fe3db574 P 19ea341ea26043299e01c85bba6b77f1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:12:10.002328 18348 raft_consensus.cc:482] T ffa7a5c99dd3474cac851118fe3db574 P 19ea341ea26043299e01c85bba6b77f1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:12:10.002432 18348 raft_consensus.cc:3037] T ffa7a5c99dd3474cac851118fe3db574 P 19ea341ea26043299e01c85bba6b77f1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:10.002475 18344 raft_consensus.cc:504] T b58941a606884f99886ac84b786b2add P d90f81c6cc4049df9f4ea38dc5fd6308 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d90f81c6cc4049df9f4ea38dc5fd6308" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 38429 } }
I20240503 14:12:10.002700 18344 leader_election.cc:302] T b58941a606884f99886ac84b786b2add P d90f81c6cc4049df9f4ea38dc5fd6308 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d90f81c6cc4049df9f4ea38dc5fd6308; no voters:
I20240503 14:12:10.003158 18345 raft_consensus.cc:388] T e21fe982cace43169ce62a2ce91d88a5 P 0c3fed516d2349dbaef0ef9e25690283 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:12:10.003283 18345 raft_consensus.cc:482] T e21fe982cace43169ce62a2ce91d88a5 P 0c3fed516d2349dbaef0ef9e25690283 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:12:10.003419 18350 raft_consensus.cc:2781] T b58941a606884f99886ac84b786b2add P d90f81c6cc4049df9f4ea38dc5fd6308 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:10.003546 18345 raft_consensus.cc:3037] T e21fe982cace43169ce62a2ce91d88a5 P 0c3fed516d2349dbaef0ef9e25690283 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:10.003942 18350 raft_consensus.cc:686] T b58941a606884f99886ac84b786b2add P d90f81c6cc4049df9f4ea38dc5fd6308 [term 1 LEADER]: Becoming Leader. State: Replica: d90f81c6cc4049df9f4ea38dc5fd6308, State: Running, Role: LEADER
I20240503 14:12:10.004305 18217 heartbeater.cc:503] Master 127.16.94.188:39819 was elected leader, sending a full tablet report...
I20240503 14:12:10.004310 18350 consensus_queue.cc:237] T b58941a606884f99886ac84b786b2add P d90f81c6cc4049df9f4ea38dc5fd6308 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d90f81c6cc4049df9f4ea38dc5fd6308" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 38429 } }
I20240503 14:12:10.005744 18348 raft_consensus.cc:504] T ffa7a5c99dd3474cac851118fe3db574 P 19ea341ea26043299e01c85bba6b77f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19ea341ea26043299e01c85bba6b77f1" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39743 } }
I20240503 14:12:10.006040 18345 raft_consensus.cc:504] T e21fe982cace43169ce62a2ce91d88a5 P 0c3fed516d2349dbaef0ef9e25690283 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c3fed516d2349dbaef0ef9e25690283" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 46657 } }
I20240503 14:12:10.006078 18348 leader_election.cc:302] T ffa7a5c99dd3474cac851118fe3db574 P 19ea341ea26043299e01c85bba6b77f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 19ea341ea26043299e01c85bba6b77f1; no voters:
I20240503 14:12:10.003443 18344 leader_election.cc:288] T b58941a606884f99886ac84b786b2add P d90f81c6cc4049df9f4ea38dc5fd6308 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:12:10.006665 18344 ts_tablet_manager.cc:1398] T b58941a606884f99886ac84b786b2add P d90f81c6cc4049df9f4ea38dc5fd6308: Time spent starting tablet: real 0.008s user 0.003s sys 0.002s
I20240503 14:12:10.006808 18348 leader_election.cc:288] T ffa7a5c99dd3474cac851118fe3db574 P 19ea341ea26043299e01c85bba6b77f1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:12:10.007028 18345 leader_election.cc:302] T e21fe982cace43169ce62a2ce91d88a5 P 0c3fed516d2349dbaef0ef9e25690283 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c3fed516d2349dbaef0ef9e25690283; no voters:
I20240503 14:12:10.007103 18353 raft_consensus.cc:2781] T ffa7a5c99dd3474cac851118fe3db574 P 19ea341ea26043299e01c85bba6b77f1 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:10.010951 18355 raft_consensus.cc:2781] T e21fe982cace43169ce62a2ce91d88a5 P 0c3fed516d2349dbaef0ef9e25690283 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:10.011649 18345 leader_election.cc:288] T e21fe982cace43169ce62a2ce91d88a5 P 0c3fed516d2349dbaef0ef9e25690283 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:12:10.012462 18296 heartbeater.cc:503] Master 127.16.94.188:39819 was elected leader, sending a full tablet report...
I20240503 14:12:10.012640 18348 ts_tablet_manager.cc:1398] T ffa7a5c99dd3474cac851118fe3db574 P 19ea341ea26043299e01c85bba6b77f1: Time spent starting tablet: real 0.011s user 0.007s sys 0.003s
I20240503 14:12:10.013456 17988 catalog_manager.cc:5484] T b58941a606884f99886ac84b786b2add P d90f81c6cc4049df9f4ea38dc5fd6308 reported cstate change: term changed from 0 to 1, leader changed from <none> to d90f81c6cc4049df9f4ea38dc5fd6308 (127.16.94.130). New cstate: current_term: 1 leader_uuid: "d90f81c6cc4049df9f4ea38dc5fd6308" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d90f81c6cc4049df9f4ea38dc5fd6308" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 38429 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:10.014829 18355 raft_consensus.cc:686] T e21fe982cace43169ce62a2ce91d88a5 P 0c3fed516d2349dbaef0ef9e25690283 [term 1 LEADER]: Becoming Leader. State: Replica: 0c3fed516d2349dbaef0ef9e25690283, State: Running, Role: LEADER
I20240503 14:12:10.015233 18355 consensus_queue.cc:237] T e21fe982cace43169ce62a2ce91d88a5 P 0c3fed516d2349dbaef0ef9e25690283 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c3fed516d2349dbaef0ef9e25690283" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 46657 } }
I20240503 14:12:10.023947 18353 raft_consensus.cc:686] T ffa7a5c99dd3474cac851118fe3db574 P 19ea341ea26043299e01c85bba6b77f1 [term 1 LEADER]: Becoming Leader. State: Replica: 19ea341ea26043299e01c85bba6b77f1, State: Running, Role: LEADER
I20240503 14:12:10.024360 18353 consensus_queue.cc:237] T ffa7a5c99dd3474cac851118fe3db574 P 19ea341ea26043299e01c85bba6b77f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19ea341ea26043299e01c85bba6b77f1" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39743 } }
I20240503 14:12:10.027907 17987 catalog_manager.cc:5484] T e21fe982cace43169ce62a2ce91d88a5 P 0c3fed516d2349dbaef0ef9e25690283 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c3fed516d2349dbaef0ef9e25690283 (127.16.94.129). New cstate: current_term: 1 leader_uuid: "0c3fed516d2349dbaef0ef9e25690283" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c3fed516d2349dbaef0ef9e25690283" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 46657 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:10.028807 18345 ts_tablet_manager.cc:1398] T e21fe982cace43169ce62a2ce91d88a5 P 0c3fed516d2349dbaef0ef9e25690283: Time spent starting tablet: real 0.031s user 0.010s sys 0.004s
I20240503 14:12:10.047557 17988 catalog_manager.cc:5484] T ffa7a5c99dd3474cac851118fe3db574 P 19ea341ea26043299e01c85bba6b77f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 19ea341ea26043299e01c85bba6b77f1 (127.16.94.131). New cstate: current_term: 1 leader_uuid: "19ea341ea26043299e01c85bba6b77f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19ea341ea26043299e01c85bba6b77f1" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39743 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:10.072594 16762 master.cc:537] Master@127.16.94.188:39819 shutting down...
I20240503 14:12:10.079145 16762 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 61252778e375497b838aed6a503c3559 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:10.079447 16762 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 61252778e375497b838aed6a503c3559 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:10.079553 16762 tablet_replica.cc:330] T 00000000000000000000000000000000 P 61252778e375497b838aed6a503c3559: stopping tablet replica
I20240503 14:12:10.084731 16762 master.cc:552] Master@127.16.94.188:39819 shutdown complete.
W20240503 14:12:11.029542 18217 heartbeater.cc:644] Failed to heartbeat to 127.16.94.188:39819 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.94.188:39819: connect: Connection refused (error 111)
I20240503 14:12:11.553099 18369 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a6eda5ef7244439b207e95121ecaa54 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 61252778e375497b838aed6a503c3559)
I20240503 14:12:11.553290 18369 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a6eda5ef7244439b207e95121ecaa54 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6eda5ef7244439b207e95121ecaa54" member_type: VOTER last_known_addr { host: "127.16.94.190" port: 43583 } } peers { permanent_uuid: "54e8c1e857c0443caea6bc05c1ce71e6" member_type: VOTER last_known_addr { host: "127.16.94.189" port: 35539 } } peers { permanent_uuid: "61252778e375497b838aed6a503c3559" member_type: VOTER last_known_addr { host: "127.16.94.188" port: 39819 } }
I20240503 14:12:11.553995 18370 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54e8c1e857c0443caea6bc05c1ce71e6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 61252778e375497b838aed6a503c3559)
I20240503 14:12:11.554158 18369 leader_election.cc:288] T 00000000000000000000000000000000 P 7a6eda5ef7244439b207e95121ecaa54 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 54e8c1e857c0443caea6bc05c1ce71e6 (127.16.94.189:35539), 61252778e375497b838aed6a503c3559 (127.16.94.188:39819)
I20240503 14:12:11.554163 18370 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54e8c1e857c0443caea6bc05c1ce71e6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6eda5ef7244439b207e95121ecaa54" member_type: VOTER last_known_addr { host: "127.16.94.190" port: 43583 } } peers { permanent_uuid: "54e8c1e857c0443caea6bc05c1ce71e6" member_type: VOTER last_known_addr { host: "127.16.94.189" port: 35539 } } peers { permanent_uuid: "61252778e375497b838aed6a503c3559" member_type: VOTER last_known_addr { host: "127.16.94.188" port: 39819 } }
I20240503 14:12:11.554601 17933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7a6eda5ef7244439b207e95121ecaa54" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "54e8c1e857c0443caea6bc05c1ce71e6" is_pre_election: true
I20240503 14:12:11.554778 17933 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 54e8c1e857c0443caea6bc05c1ce71e6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a6eda5ef7244439b207e95121ecaa54 in term 1.
I20240503 14:12:11.555171 18370 leader_election.cc:288] T 00000000000000000000000000000000 P 54e8c1e857c0443caea6bc05c1ce71e6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a6eda5ef7244439b207e95121ecaa54 (127.16.94.190:43583), 61252778e375497b838aed6a503c3559 (127.16.94.188:39819)
I20240503 14:12:11.555269 17868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "54e8c1e857c0443caea6bc05c1ce71e6" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "7a6eda5ef7244439b207e95121ecaa54" is_pre_election: true
I20240503 14:12:11.555437 17868 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7a6eda5ef7244439b207e95121ecaa54 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54e8c1e857c0443caea6bc05c1ce71e6 in term 1.
I20240503 14:12:11.555400 17844 leader_election.cc:302] T 00000000000000000000000000000000 P 7a6eda5ef7244439b207e95121ecaa54 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54e8c1e857c0443caea6bc05c1ce71e6, 7a6eda5ef7244439b207e95121ecaa54; no voters:
I20240503 14:12:11.555900 17907 leader_election.cc:302] T 00000000000000000000000000000000 P 54e8c1e857c0443caea6bc05c1ce71e6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54e8c1e857c0443caea6bc05c1ce71e6, 7a6eda5ef7244439b207e95121ecaa54; no voters:
I20240503 14:12:11.556180 18370 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 54e8c1e857c0443caea6bc05c1ce71e6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:12:11.556288 18370 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54e8c1e857c0443caea6bc05c1ce71e6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 61252778e375497b838aed6a503c3559)
I20240503 14:12:11.556394 18369 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a6eda5ef7244439b207e95121ecaa54 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:12:11.556396 18370 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54e8c1e857c0443caea6bc05c1ce71e6 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:12:11.556525 18369 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a6eda5ef7244439b207e95121ecaa54 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 61252778e375497b838aed6a503c3559)
I20240503 14:12:11.556684 18369 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a6eda5ef7244439b207e95121ecaa54 [term 1 FOLLOWER]: Advancing to term 2
W20240503 14:12:11.556886 17843 leader_election.cc:334] T 00000000000000000000000000000000 P 7a6eda5ef7244439b207e95121ecaa54 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 61252778e375497b838aed6a503c3559 (127.16.94.188:39819): Network error: Client connection negotiation failed: client connection to 127.16.94.188:39819: connect: Connection refused (error 111)
W20240503 14:12:11.557838 17908 leader_election.cc:334] T 00000000000000000000000000000000 P 54e8c1e857c0443caea6bc05c1ce71e6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 61252778e375497b838aed6a503c3559 (127.16.94.188:39819): Network error: Client connection negotiation failed: client connection to 127.16.94.188:39819: connect: Connection refused (error 111)
I20240503 14:12:11.558895 18370 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54e8c1e857c0443caea6bc05c1ce71e6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6eda5ef7244439b207e95121ecaa54" member_type: VOTER last_known_addr { host: "127.16.94.190" port: 43583 } } peers { permanent_uuid: "54e8c1e857c0443caea6bc05c1ce71e6" member_type: VOTER last_known_addr { host: "127.16.94.189" port: 35539 } } peers { permanent_uuid: "61252778e375497b838aed6a503c3559" member_type: VOTER last_known_addr { host: "127.16.94.188" port: 39819 } }
I20240503 14:12:11.558943 18369 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a6eda5ef7244439b207e95121ecaa54 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6eda5ef7244439b207e95121ecaa54" member_type: VOTER last_known_addr { host: "127.16.94.190" port: 43583 } } peers { permanent_uuid: "54e8c1e857c0443caea6bc05c1ce71e6" member_type: VOTER last_known_addr { host: "127.16.94.189" port: 35539 } } peers { permanent_uuid: "61252778e375497b838aed6a503c3559" member_type: VOTER last_known_addr { host: "127.16.94.188" port: 39819 } }
I20240503 14:12:11.559360 18370 leader_election.cc:288] T 00000000000000000000000000000000 P 54e8c1e857c0443caea6bc05c1ce71e6 [CANDIDATE]: Term 2 election: Requested vote from peers 7a6eda5ef7244439b207e95121ecaa54 (127.16.94.190:43583), 61252778e375497b838aed6a503c3559 (127.16.94.188:39819)
I20240503 14:12:11.559635 18369 leader_election.cc:288] T 00000000000000000000000000000000 P 7a6eda5ef7244439b207e95121ecaa54 [CANDIDATE]: Term 2 election: Requested vote from peers 54e8c1e857c0443caea6bc05c1ce71e6 (127.16.94.189:35539), 61252778e375497b838aed6a503c3559 (127.16.94.188:39819)
I20240503 14:12:11.559973 17933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7a6eda5ef7244439b207e95121ecaa54" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "54e8c1e857c0443caea6bc05c1ce71e6"
I20240503 14:12:11.560185 17933 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 54e8c1e857c0443caea6bc05c1ce71e6 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a6eda5ef7244439b207e95121ecaa54 in current term 2: Already voted for candidate 54e8c1e857c0443caea6bc05c1ce71e6 in this term.
I20240503 14:12:11.560132 17868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "54e8c1e857c0443caea6bc05c1ce71e6" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "7a6eda5ef7244439b207e95121ecaa54"
I20240503 14:12:11.560325 17868 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 7a6eda5ef7244439b207e95121ecaa54 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 54e8c1e857c0443caea6bc05c1ce71e6 in current term 2: Already voted for candidate 7a6eda5ef7244439b207e95121ecaa54 in this term.
W20240503 14:12:11.560891 17843 leader_election.cc:334] T 00000000000000000000000000000000 P 7a6eda5ef7244439b207e95121ecaa54 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 61252778e375497b838aed6a503c3559 (127.16.94.188:39819): Network error: Client connection negotiation failed: client connection to 127.16.94.188:39819: connect: Connection refused (error 111)
I20240503 14:12:11.561012 17843 leader_election.cc:302] T 00000000000000000000000000000000 P 7a6eda5ef7244439b207e95121ecaa54 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a6eda5ef7244439b207e95121ecaa54; no voters: 54e8c1e857c0443caea6bc05c1ce71e6, 61252778e375497b838aed6a503c3559
I20240503 14:12:11.561271 18369 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 7a6eda5ef7244439b207e95121ecaa54 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20240503 14:12:11.561358 17908 leader_election.cc:334] T 00000000000000000000000000000000 P 54e8c1e857c0443caea6bc05c1ce71e6 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 61252778e375497b838aed6a503c3559 (127.16.94.188:39819): Network error: Client connection negotiation failed: client connection to 127.16.94.188:39819: connect: Connection refused (error 111)
I20240503 14:12:11.561488 17908 leader_election.cc:302] T 00000000000000000000000000000000 P 54e8c1e857c0443caea6bc05c1ce71e6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 54e8c1e857c0443caea6bc05c1ce71e6; no voters: 61252778e375497b838aed6a503c3559, 7a6eda5ef7244439b207e95121ecaa54
I20240503 14:12:11.561753 18370 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 54e8c1e857c0443caea6bc05c1ce71e6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240503 14:12:13.134409 18381 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54e8c1e857c0443caea6bc05c1ce71e6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:13.134606 18381 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54e8c1e857c0443caea6bc05c1ce71e6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6eda5ef7244439b207e95121ecaa54" member_type: VOTER last_known_addr { host: "127.16.94.190" port: 43583 } } peers { permanent_uuid: "54e8c1e857c0443caea6bc05c1ce71e6" member_type: VOTER last_known_addr { host: "127.16.94.189" port: 35539 } } peers { permanent_uuid: "61252778e375497b838aed6a503c3559" member_type: VOTER last_known_addr { host: "127.16.94.188" port: 39819 } }
I20240503 14:12:13.135308 18381 leader_election.cc:288] T 00000000000000000000000000000000 P 54e8c1e857c0443caea6bc05c1ce71e6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7a6eda5ef7244439b207e95121ecaa54 (127.16.94.190:43583), 61252778e375497b838aed6a503c3559 (127.16.94.188:39819)
I20240503 14:12:13.135819 17868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "54e8c1e857c0443caea6bc05c1ce71e6" candidate_term: 3 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "7a6eda5ef7244439b207e95121ecaa54" is_pre_election: true
I20240503 14:12:13.136036 17868 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7a6eda5ef7244439b207e95121ecaa54 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54e8c1e857c0443caea6bc05c1ce71e6 in term 2.
W20240503 14:12:13.136234 17908 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.94.188:39819: connect: Connection refused (error 111) [suppressed 14 similar messages]
I20240503 14:12:13.136479 17907 leader_election.cc:302] T 00000000000000000000000000000000 P 54e8c1e857c0443caea6bc05c1ce71e6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54e8c1e857c0443caea6bc05c1ce71e6, 7a6eda5ef7244439b207e95121ecaa54; no voters:
I20240503 14:12:13.136736 18381 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 54e8c1e857c0443caea6bc05c1ce71e6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240503 14:12:13.136890 18381 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54e8c1e857c0443caea6bc05c1ce71e6 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:13.136993 18381 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54e8c1e857c0443caea6bc05c1ce71e6 [term 2 FOLLOWER]: Advancing to term 3
W20240503 14:12:13.137687 17908 leader_election.cc:334] T 00000000000000000000000000000000 P 54e8c1e857c0443caea6bc05c1ce71e6 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 61252778e375497b838aed6a503c3559 (127.16.94.188:39819): Network error: Client connection negotiation failed: client connection to 127.16.94.188:39819: connect: Connection refused (error 111)
I20240503 14:12:13.139151 18381 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54e8c1e857c0443caea6bc05c1ce71e6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6eda5ef7244439b207e95121ecaa54" member_type: VOTER last_known_addr { host: "127.16.94.190" port: 43583 } } peers { permanent_uuid: "54e8c1e857c0443caea6bc05c1ce71e6" member_type: VOTER last_known_addr { host: "127.16.94.189" port: 35539 } } peers { permanent_uuid: "61252778e375497b838aed6a503c3559" member_type: VOTER last_known_addr { host: "127.16.94.188" port: 39819 } }
I20240503 14:12:13.139600 18381 leader_election.cc:288] T 00000000000000000000000000000000 P 54e8c1e857c0443caea6bc05c1ce71e6 [CANDIDATE]: Term 3 election: Requested vote from peers 7a6eda5ef7244439b207e95121ecaa54 (127.16.94.190:43583), 61252778e375497b838aed6a503c3559 (127.16.94.188:39819)
I20240503 14:12:13.140012 17868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "54e8c1e857c0443caea6bc05c1ce71e6" candidate_term: 3 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "7a6eda5ef7244439b207e95121ecaa54"
I20240503 14:12:13.140172 17868 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a6eda5ef7244439b207e95121ecaa54 [term 2 FOLLOWER]: Advancing to term 3
W20240503 14:12:13.140779 17908 leader_election.cc:334] T 00000000000000000000000000000000 P 54e8c1e857c0443caea6bc05c1ce71e6 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 61252778e375497b838aed6a503c3559 (127.16.94.188:39819): Network error: Client connection negotiation failed: client connection to 127.16.94.188:39819: connect: Connection refused (error 111)
I20240503 14:12:13.142074 17868 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7a6eda5ef7244439b207e95121ecaa54 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54e8c1e857c0443caea6bc05c1ce71e6 in term 3.
I20240503 14:12:13.142446 17907 leader_election.cc:302] T 00000000000000000000000000000000 P 54e8c1e857c0443caea6bc05c1ce71e6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54e8c1e857c0443caea6bc05c1ce71e6, 7a6eda5ef7244439b207e95121ecaa54; no voters: 61252778e375497b838aed6a503c3559
I20240503 14:12:13.142710 18381 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 54e8c1e857c0443caea6bc05c1ce71e6 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 14:12:13.143060 18381 raft_consensus.cc:686] T 00000000000000000000000000000000 P 54e8c1e857c0443caea6bc05c1ce71e6 [term 3 LEADER]: Becoming Leader. State: Replica: 54e8c1e857c0443caea6bc05c1ce71e6, State: Running, Role: LEADER
I20240503 14:12:13.143339 18381 consensus_queue.cc:237] T 00000000000000000000000000000000 P 54e8c1e857c0443caea6bc05c1ce71e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6eda5ef7244439b207e95121ecaa54" member_type: VOTER last_known_addr { host: "127.16.94.190" port: 43583 } } peers { permanent_uuid: "54e8c1e857c0443caea6bc05c1ce71e6" member_type: VOTER last_known_addr { host: "127.16.94.189" port: 35539 } } peers { permanent_uuid: "61252778e375497b838aed6a503c3559" member_type: VOTER last_known_addr { host: "127.16.94.188" port: 39819 } }
I20240503 14:12:13.144667 18384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54e8c1e857c0443caea6bc05c1ce71e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54e8c1e857c0443caea6bc05c1ce71e6. Latest consensus state: current_term: 3 leader_uuid: "54e8c1e857c0443caea6bc05c1ce71e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6eda5ef7244439b207e95121ecaa54" member_type: VOTER last_known_addr { host: "127.16.94.190" port: 43583 } } peers { permanent_uuid: "54e8c1e857c0443caea6bc05c1ce71e6" member_type: VOTER last_known_addr { host: "127.16.94.189" port: 35539 } } peers { permanent_uuid: "61252778e375497b838aed6a503c3559" member_type: VOTER last_known_addr { host: "127.16.94.188" port: 39819 } } }
I20240503 14:12:13.144850 18384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54e8c1e857c0443caea6bc05c1ce71e6 [sys.catalog]: This master's current role is: LEADER
I20240503 14:12:13.145308 18386 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 14:12:13.146978 18386 catalog_manager.cc:669] Loaded metadata for table test-workload [id=87bb46fdb45d4757ae04513926b680b5]
I20240503 14:12:13.148265 18386 tablet_loader.cc:96] loaded metadata for tablet b58941a606884f99886ac84b786b2add (table test-workload [id=87bb46fdb45d4757ae04513926b680b5])
I20240503 14:12:13.148579 18386 tablet_loader.cc:96] loaded metadata for tablet e21fe982cace43169ce62a2ce91d88a5 (table test-workload [id=87bb46fdb45d4757ae04513926b680b5])
I20240503 14:12:13.148861 18386 tablet_loader.cc:96] loaded metadata for tablet ffa7a5c99dd3474cac851118fe3db574 (table test-workload [id=87bb46fdb45d4757ae04513926b680b5])
I20240503 14:12:13.149060 18386 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 14:12:13.149911 18386 catalog_manager.cc:1259] Loaded cluster ID: e03a07a9169d4531a47a50f9a0e1c699
I20240503 14:12:13.150000 18386 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 14:12:13.151129 18386 catalog_manager.cc:1504] Loading token signing keys...
I20240503 14:12:13.151872 18386 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 54e8c1e857c0443caea6bc05c1ce71e6: Loaded TSK: 0
I20240503 14:12:13.152465 18386 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 14:12:13.529379 17868 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7a6eda5ef7244439b207e95121ecaa54 [term 3 FOLLOWER]: Refusing update from remote peer 54e8c1e857c0443caea6bc05c1ce71e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20240503 14:12:13.529995 18384 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 54e8c1e857c0443caea6bc05c1ce71e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6eda5ef7244439b207e95121ecaa54" member_type: VOTER last_known_addr { host: "127.16.94.190" port: 43583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 14:12:13.531672 18387 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a6eda5ef7244439b207e95121ecaa54 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54e8c1e857c0443caea6bc05c1ce71e6. Latest consensus state: current_term: 3 leader_uuid: "54e8c1e857c0443caea6bc05c1ce71e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6eda5ef7244439b207e95121ecaa54" member_type: VOTER last_known_addr { host: "127.16.94.190" port: 43583 } } peers { permanent_uuid: "54e8c1e857c0443caea6bc05c1ce71e6" member_type: VOTER last_known_addr { host: "127.16.94.189" port: 35539 } } peers { permanent_uuid: "61252778e375497b838aed6a503c3559" member_type: VOTER last_known_addr { host: "127.16.94.188" port: 39819 } } }
I20240503 14:12:13.531909 18387 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a6eda5ef7244439b207e95121ecaa54 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 14:12:13.533195 18384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54e8c1e857c0443caea6bc05c1ce71e6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "54e8c1e857c0443caea6bc05c1ce71e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6eda5ef7244439b207e95121ecaa54" member_type: VOTER last_known_addr { host: "127.16.94.190" port: 43583 } } peers { permanent_uuid: "54e8c1e857c0443caea6bc05c1ce71e6" member_type: VOTER last_known_addr { host: "127.16.94.189" port: 35539 } } peers { permanent_uuid: "61252778e375497b838aed6a503c3559" member_type: VOTER last_known_addr { host: "127.16.94.188" port: 39819 } } }
I20240503 14:12:13.533504 18384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54e8c1e857c0443caea6bc05c1ce71e6 [sys.catalog]: This master's current role is: LEADER
I20240503 14:12:13.533946 18387 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a6eda5ef7244439b207e95121ecaa54 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "54e8c1e857c0443caea6bc05c1ce71e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6eda5ef7244439b207e95121ecaa54" member_type: VOTER last_known_addr { host: "127.16.94.190" port: 43583 } } peers { permanent_uuid: "54e8c1e857c0443caea6bc05c1ce71e6" member_type: VOTER last_known_addr { host: "127.16.94.189" port: 35539 } } peers { permanent_uuid: "61252778e375497b838aed6a503c3559" member_type: VOTER last_known_addr { host: "127.16.94.188" port: 39819 } } }
I20240503 14:12:13.534158 18387 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a6eda5ef7244439b207e95121ecaa54 [sys.catalog]: This master's current role is: FOLLOWER
W20240503 14:12:13.534467 17908 consensus_peers.cc:485] T 00000000000000000000000000000000 P 54e8c1e857c0443caea6bc05c1ce71e6 -> Peer 61252778e375497b838aed6a503c3559 (127.16.94.188:39819): Couldn't send request to peer 61252778e375497b838aed6a503c3559. Status: Network error: Client connection negotiation failed: client connection to 127.16.94.188:39819: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:13.924057 18053 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:12:13.924175 18053 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:12:14.027921 18219 heartbeater.cc:503] Master 127.16.94.189:35539 was elected leader, sending a full tablet report...
I20240503 14:12:14.058089 18298 heartbeater.cc:503] Master 127.16.94.189:35539 was elected leader, sending a full tablet report...
I20240503 14:12:14.250388 16762 tablet_server.cc:178] TabletServer@127.16.94.129:0 shutting down...
I20240503 14:12:14.258424 16762 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 14:12:14.258721 16762 tablet_replica.cc:330] T e21fe982cace43169ce62a2ce91d88a5 P 0c3fed516d2349dbaef0ef9e25690283: stopping tablet replica
I20240503 14:12:14.258896 16762 raft_consensus.cc:2229] T e21fe982cace43169ce62a2ce91d88a5 P 0c3fed516d2349dbaef0ef9e25690283 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:14.259063 16762 raft_consensus.cc:2258] T e21fe982cace43169ce62a2ce91d88a5 P 0c3fed516d2349dbaef0ef9e25690283 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:14.272483 16762 tablet_server.cc:194] TabletServer@127.16.94.129:0 shutdown complete.
I20240503 14:12:14.275331 16762 tablet_server.cc:178] TabletServer@127.16.94.130:0 shutting down...
I20240503 14:12:14.284215 16762 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 14:12:14.284492 16762 tablet_replica.cc:330] T b58941a606884f99886ac84b786b2add P d90f81c6cc4049df9f4ea38dc5fd6308: stopping tablet replica
I20240503 14:12:14.284674 16762 raft_consensus.cc:2229] T b58941a606884f99886ac84b786b2add P d90f81c6cc4049df9f4ea38dc5fd6308 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:14.284844 16762 raft_consensus.cc:2258] T b58941a606884f99886ac84b786b2add P d90f81c6cc4049df9f4ea38dc5fd6308 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:14.299726 16762 tablet_server.cc:194] TabletServer@127.16.94.130:0 shutdown complete.
I20240503 14:12:14.302423 16762 tablet_server.cc:178] TabletServer@127.16.94.131:0 shutting down...
I20240503 14:12:14.310978 16762 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 14:12:14.311282 16762 tablet_replica.cc:330] T ffa7a5c99dd3474cac851118fe3db574 P 19ea341ea26043299e01c85bba6b77f1: stopping tablet replica
I20240503 14:12:14.311472 16762 raft_consensus.cc:2229] T ffa7a5c99dd3474cac851118fe3db574 P 19ea341ea26043299e01c85bba6b77f1 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:14.311681 16762 raft_consensus.cc:2258] T ffa7a5c99dd3474cac851118fe3db574 P 19ea341ea26043299e01c85bba6b77f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:14.325338 16762 tablet_server.cc:194] TabletServer@127.16.94.131:0 shutdown complete.
I20240503 14:12:14.328351 16762 master.cc:537] Master@127.16.94.190:43583 shutting down...
I20240503 14:12:14.335235 16762 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 7a6eda5ef7244439b207e95121ecaa54 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:14.335480 16762 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 7a6eda5ef7244439b207e95121ecaa54 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:14.335613 16762 tablet_replica.cc:330] T 00000000000000000000000000000000 P 7a6eda5ef7244439b207e95121ecaa54: stopping tablet replica
I20240503 14:12:14.356629 16762 master.cc:552] Master@127.16.94.190:43583 shutdown complete.
I20240503 14:12:14.360929 16762 master.cc:537] Master@127.16.94.189:35539 shutting down...
I20240503 14:12:14.368134 16762 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 54e8c1e857c0443caea6bc05c1ce71e6 [term 3 LEADER]: Raft consensus shutting down.
I20240503 14:12:14.368503 16762 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 54e8c1e857c0443caea6bc05c1ce71e6 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:14.368613 16762 tablet_replica.cc:330] T 00000000000000000000000000000000 P 54e8c1e857c0443caea6bc05c1ce71e6: stopping tablet replica
I20240503 14:12:14.381914 16762 master.cc:552] Master@127.16.94.189:35539 shutdown complete.
[ OK ] AutoRebalancerTest.NextLeaderResumesAutoRebalancing (6711 ms)
[ RUN ] AutoRebalancerTest.MovesScheduledIfAddTserver
I20240503 14:12:14.394045 16762 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.16.94.190:44641
I20240503 14:12:14.394551 16762 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:14.397390 18395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:14.398260 18396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:14.398955 18398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:14.399667 16762 server_base.cc:1034] running on GCE node
I20240503 14:12:14.399921 16762 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:12:14.399974 16762 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:12:14.400022 16762 hybrid_clock.cc:648] HybridClock initialized: now 1714745534400018 us; error 0 us; skew 500 ppm
I20240503 14:12:14.400162 16762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:14.405640 16762 webserver.cc:460] Webserver started at http://127.16.94.190:37775/ using document root <none> and password file <none>
I20240503 14:12:14.405898 16762 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:14.405961 16762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:14.406199 16762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:14.406952 16762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714745515241994-16762-0/minicluster-data/master-0-root/instance:
uuid: "c157bc32c1144102a7794076facb2b25"
format_stamp: "Formatted at 2024-05-03 14:12:14 on dist-test-slave-v27g"
I20240503 14:12:14.409428 16762 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20240503 14:12:14.411060 18403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:14.411370 16762 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240503 14:12:14.411464 16762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714745515241994-16762-0/minicluster-data/master-0-root
uuid: "c157bc32c1144102a7794076facb2b25"
format_stamp: "Formatted at 2024-05-03 14:12:14 on dist-test-slave-v27g"
I20240503 14:12:14.411572 16762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714745515241994-16762-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714745515241994-16762-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714745515241994-16762-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:14.434728 16762 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:14.435315 16762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:14.446918 16762 rpc_server.cc:307] RPC server started. Bound to: 127.16.94.190:44641
I20240503 14:12:14.446928 18454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.94.190:44641 every 8 connection(s)
I20240503 14:12:14.448165 18455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.452054 18455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c157bc32c1144102a7794076facb2b25: Bootstrap starting.
I20240503 14:12:14.453135 18455 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c157bc32c1144102a7794076facb2b25: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:14.454535 18455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c157bc32c1144102a7794076facb2b25: No bootstrap required, opened a new log
I20240503 14:12:14.455034 18455 raft_consensus.cc:348] T 00000000000000000000000000000000 P c157bc32c1144102a7794076facb2b25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c157bc32c1144102a7794076facb2b25" member_type: VOTER }
I20240503 14:12:14.455143 18455 raft_consensus.cc:374] T 00000000000000000000000000000000 P c157bc32c1144102a7794076facb2b25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:14.455179 18455 raft_consensus.cc:729] T 00000000000000000000000000000000 P c157bc32c1144102a7794076facb2b25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c157bc32c1144102a7794076facb2b25, State: Initialized, Role: FOLLOWER
I20240503 14:12:14.455308 18455 consensus_queue.cc:260] T 00000000000000000000000000000000 P c157bc32c1144102a7794076facb2b25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c157bc32c1144102a7794076facb2b25" member_type: VOTER }
I20240503 14:12:14.455390 18455 raft_consensus.cc:388] T 00000000000000000000000000000000 P c157bc32c1144102a7794076facb2b25 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:12:14.455425 18455 raft_consensus.cc:482] T 00000000000000000000000000000000 P c157bc32c1144102a7794076facb2b25 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:12:14.455469 18455 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c157bc32c1144102a7794076facb2b25 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:14.457021 18455 raft_consensus.cc:504] T 00000000000000000000000000000000 P c157bc32c1144102a7794076facb2b25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c157bc32c1144102a7794076facb2b25" member_type: VOTER }
I20240503 14:12:14.457194 18455 leader_election.cc:302] T 00000000000000000000000000000000 P c157bc32c1144102a7794076facb2b25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c157bc32c1144102a7794076facb2b25; no voters:
I20240503 14:12:14.457489 18455 leader_election.cc:288] T 00000000000000000000000000000000 P c157bc32c1144102a7794076facb2b25 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:12:14.457626 18458 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c157bc32c1144102a7794076facb2b25 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:14.458101 18458 raft_consensus.cc:686] T 00000000000000000000000000000000 P c157bc32c1144102a7794076facb2b25 [term 1 LEADER]: Becoming Leader. State: Replica: c157bc32c1144102a7794076facb2b25, State: Running, Role: LEADER
I20240503 14:12:14.458353 18455 sys_catalog.cc:564] T 00000000000000000000000000000000 P c157bc32c1144102a7794076facb2b25 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 14:12:14.458314 18458 consensus_queue.cc:237] T 00000000000000000000000000000000 P c157bc32c1144102a7794076facb2b25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c157bc32c1144102a7794076facb2b25" member_type: VOTER }
I20240503 14:12:14.459002 18460 sys_catalog.cc:455] T 00000000000000000000000000000000 P c157bc32c1144102a7794076facb2b25 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c157bc32c1144102a7794076facb2b25. Latest consensus state: current_term: 1 leader_uuid: "c157bc32c1144102a7794076facb2b25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c157bc32c1144102a7794076facb2b25" member_type: VOTER } }
I20240503 14:12:14.458957 18459 sys_catalog.cc:455] T 00000000000000000000000000000000 P c157bc32c1144102a7794076facb2b25 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c157bc32c1144102a7794076facb2b25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c157bc32c1144102a7794076facb2b25" member_type: VOTER } }
I20240503 14:12:14.459131 18460 sys_catalog.cc:458] T 00000000000000000000000000000000 P c157bc32c1144102a7794076facb2b25 [sys.catalog]: This master's current role is: LEADER
I20240503 14:12:14.459160 18459 sys_catalog.cc:458] T 00000000000000000000000000000000 P c157bc32c1144102a7794076facb2b25 [sys.catalog]: This master's current role is: LEADER
I20240503 14:12:14.459630 18462 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 14:12:14.460943 18462 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 14:12:14.463670 16762 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 14:12:14.463739 18462 catalog_manager.cc:1347] Generated new cluster ID: f6960b7d380f4459b3b7c83012d2ed8c
I20240503 14:12:14.463804 18462 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 14:12:14.476105 18462 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 14:12:14.476773 18462 catalog_manager.cc:1504] Loading token signing keys...
I20240503 14:12:14.482414 18462 catalog_manager.cc:5857] T 00000000000000000000000000000000 P c157bc32c1144102a7794076facb2b25: Generated new TSK 0
I20240503 14:12:14.482621 18462 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 14:12:14.496104 16762 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:14.499050 18476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:14.499883 18477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:14.500586 18479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:14.501356 16762 server_base.cc:1034] running on GCE node
I20240503 14:12:14.501590 16762 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:12:14.501636 16762 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:12:14.501683 16762 hybrid_clock.cc:648] HybridClock initialized: now 1714745534501679 us; error 0 us; skew 500 ppm
I20240503 14:12:14.501806 16762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:14.503041 16762 webserver.cc:460] Webserver started at http://127.16.94.129:34201/ using document root <none> and password file <none>
I20240503 14:12:14.503260 16762 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:14.503324 16762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:14.503453 16762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:14.504257 16762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714745515241994-16762-0/minicluster-data/ts-0-root/instance:
uuid: "dcda0acf1e7f40369d82f940edc2becd"
format_stamp: "Formatted at 2024-05-03 14:12:14 on dist-test-slave-v27g"
I20240503 14:12:14.507627 16762 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20240503 14:12:14.510185 18484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:14.510587 16762 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:14.510689 16762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714745515241994-16762-0/minicluster-data/ts-0-root
uuid: "dcda0acf1e7f40369d82f940edc2becd"
format_stamp: "Formatted at 2024-05-03 14:12:14 on dist-test-slave-v27g"
I20240503 14:12:14.510823 16762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714745515241994-16762-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714745515241994-16762-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714745515241994-16762-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:14.526995 16762 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:14.527539 16762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:14.528086 16762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:14.528864 16762 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:14.528914 16762 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:14.528975 16762 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:14.529012 16762 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:14.540202 16762 rpc_server.cc:307] RPC server started. Bound to: 127.16.94.129:46679
I20240503 14:12:14.540223 18547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.94.129:46679 every 8 connection(s)
I20240503 14:12:14.541743 16762 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:14.544793 18552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:14.545506 18553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:14.546226 18555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:14.546957 16762 server_base.cc:1034] running on GCE node
I20240503 14:12:14.547230 16762 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:12:14.547291 16762 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:12:14.547335 16762 hybrid_clock.cc:648] HybridClock initialized: now 1714745534547332 us; error 0 us; skew 500 ppm
I20240503 14:12:14.547474 16762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:14.547811 18548 heartbeater.cc:347] Connected to a master server at 127.16.94.190:44641
I20240503 14:12:14.547919 18548 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:14.548308 18548 heartbeater.cc:511] Master 127.16.94.190:44641 requested a full tablet report, sending...
I20240503 14:12:14.548847 16762 webserver.cc:460] Webserver started at http://127.16.94.130:40217/ using document root <none> and password file <none>
I20240503 14:12:14.549046 16762 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:14.549106 16762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:14.549139 18420 ts_manager.cc:194] Registered new tserver with Master: dcda0acf1e7f40369d82f940edc2becd (127.16.94.129:46679)
I20240503 14:12:14.549216 16762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:14.549846 16762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714745515241994-16762-0/minicluster-data/ts-1-root/instance:
uuid: "f56ca8f5d65d4597a2ac8651dae8d6b7"
format_stamp: "Formatted at 2024-05-03 14:12:14 on dist-test-slave-v27g"
I20240503 14:12:14.550155 18420 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33034
I20240503 14:12:14.552102 16762 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20240503 14:12:14.553407 18560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:14.553728 16762 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240503 14:12:14.553826 16762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714745515241994-16762-0/minicluster-data/ts-1-root
uuid: "f56ca8f5d65d4597a2ac8651dae8d6b7"
format_stamp: "Formatted at 2024-05-03 14:12:14 on dist-test-slave-v27g"
I20240503 14:12:14.553913 16762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714745515241994-16762-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714745515241994-16762-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714745515241994-16762-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:14.571852 16762 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:14.572320 16762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:14.572798 16762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:14.573532 16762 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:14.573583 16762 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:14.573689 16762 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:14.573727 16762 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:14.586585 16762 rpc_server.cc:307] RPC server started. Bound to: 127.16.94.130:39207
I20240503 14:12:14.586613 18622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.94.130:39207 every 8 connection(s)
I20240503 14:12:14.588127 16762 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:14.591845 18627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:14.592310 18626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:14.593119 18629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:14.593778 16762 server_base.cc:1034] running on GCE node
I20240503 14:12:14.593775 18623 heartbeater.cc:347] Connected to a master server at 127.16.94.190:44641
I20240503 14:12:14.593945 18623 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:14.594081 16762 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:12:14.594138 16762 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:12:14.594182 16762 hybrid_clock.cc:648] HybridClock initialized: now 1714745534594178 us; error 0 us; skew 500 ppm
I20240503 14:12:14.594300 18623 heartbeater.cc:511] Master 127.16.94.190:44641 requested a full tablet report, sending...
I20240503 14:12:14.594305 16762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:14.595196 18420 ts_manager.cc:194] Registered new tserver with Master: f56ca8f5d65d4597a2ac8651dae8d6b7 (127.16.94.130:39207)
I20240503 14:12:14.595665 16762 webserver.cc:460] Webserver started at http://127.16.94.131:32957/ using document root <none> and password file <none>
I20240503 14:12:14.595878 16762 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:14.595942 16762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:14.596037 16762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:14.596144 18420 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33050
I20240503 14:12:14.596735 16762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714745515241994-16762-0/minicluster-data/ts-2-root/instance:
uuid: "e53c5f1c8f6f4e90ad01fb89fd81a05c"
format_stamp: "Formatted at 2024-05-03 14:12:14 on dist-test-slave-v27g"
I20240503 14:12:14.598923 16762 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20240503 14:12:14.600298 18634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:14.600627 16762 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 14:12:14.600724 16762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714745515241994-16762-0/minicluster-data/ts-2-root
uuid: "e53c5f1c8f6f4e90ad01fb89fd81a05c"
format_stamp: "Formatted at 2024-05-03 14:12:14 on dist-test-slave-v27g"
I20240503 14:12:14.600814 16762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714745515241994-16762-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714745515241994-16762-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714745515241994-16762-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:14.624238 16762 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:14.624722 16762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:14.625218 16762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:14.625929 16762 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:14.625982 16762 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:14.626080 16762 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:14.626125 16762 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:14.637989 16762 rpc_server.cc:307] RPC server started. Bound to: 127.16.94.131:46285
I20240503 14:12:14.638020 18697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.94.131:46285 every 8 connection(s)
I20240503 14:12:14.644385 18698 heartbeater.cc:347] Connected to a master server at 127.16.94.190:44641
I20240503 14:12:14.644505 18698 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:14.644771 18698 heartbeater.cc:511] Master 127.16.94.190:44641 requested a full tablet report, sending...
I20240503 14:12:14.645458 18420 ts_manager.cc:194] Registered new tserver with Master: e53c5f1c8f6f4e90ad01fb89fd81a05c (127.16.94.131:46285)
I20240503 14:12:14.645520 16762 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.006742143s
I20240503 14:12:14.646178 18420 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33054
I20240503 14:12:15.551192 18548 heartbeater.cc:503] Master 127.16.94.190:44641 was elected leader, sending a full tablet report...
I20240503 14:12:15.597318 18623 heartbeater.cc:503] Master 127.16.94.190:44641 was elected leader, sending a full tablet report...
I20240503 14:12:15.647276 18698 heartbeater.cc:503] Master 127.16.94.190:44641 was elected leader, sending a full tablet report...
I20240503 14:12:15.672775 16762 test_util.cc:268] Using random seed: 1317581249
I20240503 14:12:15.683394 18420 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33064:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377?""\004\001\000\377\377\377?"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:12:15.684023 18420 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:15.712639 18512 tablet_service.cc:1467] Processing CreateTablet for tablet e7995e0f8d084c659b0d702fda4acf66 (DEFAULT_TABLE table=test-workload [id=0cc3dddc1ea14c5a908bfcd1891b6419]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240503 14:12:15.712661 18511 tablet_service.cc:1467] Processing CreateTablet for tablet 340236565cbb4796ad23ed5d13c9f14e (DEFAULT_TABLE table=test-workload [id=0cc3dddc1ea14c5a908bfcd1891b6419]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240503 14:12:15.713207 18512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7995e0f8d084c659b0d702fda4acf66. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.713801 18511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 340236565cbb4796ad23ed5d13c9f14e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.715147 18662 tablet_service.cc:1467] Processing CreateTablet for tablet e7995e0f8d084c659b0d702fda4acf66 (DEFAULT_TABLE table=test-workload [id=0cc3dddc1ea14c5a908bfcd1891b6419]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240503 14:12:15.715174 18661 tablet_service.cc:1467] Processing CreateTablet for tablet 340236565cbb4796ad23ed5d13c9f14e (DEFAULT_TABLE table=test-workload [id=0cc3dddc1ea14c5a908bfcd1891b6419]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240503 14:12:15.715579 18662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7995e0f8d084c659b0d702fda4acf66. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.715831 18661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 340236565cbb4796ad23ed5d13c9f14e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.721467 18719 tablet_bootstrap.cc:492] T 340236565cbb4796ad23ed5d13c9f14e P e53c5f1c8f6f4e90ad01fb89fd81a05c: Bootstrap starting.
I20240503 14:12:15.721896 18588 tablet_service.cc:1467] Processing CreateTablet for tablet e7995e0f8d084c659b0d702fda4acf66 (DEFAULT_TABLE table=test-workload [id=0cc3dddc1ea14c5a908bfcd1891b6419]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240503 14:12:15.722314 18588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7995e0f8d084c659b0d702fda4acf66. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.722501 18587 tablet_service.cc:1467] Processing CreateTablet for tablet 340236565cbb4796ad23ed5d13c9f14e (DEFAULT_TABLE table=test-workload [id=0cc3dddc1ea14c5a908bfcd1891b6419]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240503 14:12:15.723021 18587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 340236565cbb4796ad23ed5d13c9f14e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.723999 18719 tablet_bootstrap.cc:654] T 340236565cbb4796ad23ed5d13c9f14e P e53c5f1c8f6f4e90ad01fb89fd81a05c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:15.728454 18718 tablet_bootstrap.cc:492] T e7995e0f8d084c659b0d702fda4acf66 P dcda0acf1e7f40369d82f940edc2becd: Bootstrap starting.
I20240503 14:12:15.731231 18718 tablet_bootstrap.cc:654] T e7995e0f8d084c659b0d702fda4acf66 P dcda0acf1e7f40369d82f940edc2becd: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:15.732782 18718 tablet_bootstrap.cc:492] T e7995e0f8d084c659b0d702fda4acf66 P dcda0acf1e7f40369d82f940edc2becd: No bootstrap required, opened a new log
I20240503 14:12:15.732900 18718 ts_tablet_manager.cc:1367] T e7995e0f8d084c659b0d702fda4acf66 P dcda0acf1e7f40369d82f940edc2becd: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 14:12:15.733419 18718 raft_consensus.cc:348] T e7995e0f8d084c659b0d702fda4acf66 P dcda0acf1e7f40369d82f940edc2becd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f56ca8f5d65d4597a2ac8651dae8d6b7" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39207 } } peers { permanent_uuid: "dcda0acf1e7f40369d82f940edc2becd" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 46679 } } peers { permanent_uuid: "e53c5f1c8f6f4e90ad01fb89fd81a05c" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 46285 } }
I20240503 14:12:15.733594 18718 raft_consensus.cc:374] T e7995e0f8d084c659b0d702fda4acf66 P dcda0acf1e7f40369d82f940edc2becd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:15.733672 18718 raft_consensus.cc:729] T e7995e0f8d084c659b0d702fda4acf66 P dcda0acf1e7f40369d82f940edc2becd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcda0acf1e7f40369d82f940edc2becd, State: Initialized, Role: FOLLOWER
I20240503 14:12:15.733856 18718 consensus_queue.cc:260] T e7995e0f8d084c659b0d702fda4acf66 P dcda0acf1e7f40369d82f940edc2becd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f56ca8f5d65d4597a2ac8651dae8d6b7" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39207 } } peers { permanent_uuid: "dcda0acf1e7f40369d82f940edc2becd" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 46679 } } peers { permanent_uuid: "e53c5f1c8f6f4e90ad01fb89fd81a05c" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 46285 } }
I20240503 14:12:15.734414 18721 tablet_bootstrap.cc:492] T e7995e0f8d084c659b0d702fda4acf66 P f56ca8f5d65d4597a2ac8651dae8d6b7: Bootstrap starting.
I20240503 14:12:15.734593 18718 ts_tablet_manager.cc:1398] T e7995e0f8d084c659b0d702fda4acf66 P dcda0acf1e7f40369d82f940edc2becd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:15.736642 18721 tablet_bootstrap.cc:654] T e7995e0f8d084c659b0d702fda4acf66 P f56ca8f5d65d4597a2ac8651dae8d6b7: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:15.737131 18718 tablet_bootstrap.cc:492] T 340236565cbb4796ad23ed5d13c9f14e P dcda0acf1e7f40369d82f940edc2becd: Bootstrap starting.
I20240503 14:12:15.739049 18719 tablet_bootstrap.cc:492] T 340236565cbb4796ad23ed5d13c9f14e P e53c5f1c8f6f4e90ad01fb89fd81a05c: No bootstrap required, opened a new log
I20240503 14:12:15.739190 18719 ts_tablet_manager.cc:1367] T 340236565cbb4796ad23ed5d13c9f14e P e53c5f1c8f6f4e90ad01fb89fd81a05c: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.008s
I20240503 14:12:15.739481 18718 tablet_bootstrap.cc:654] T 340236565cbb4796ad23ed5d13c9f14e P dcda0acf1e7f40369d82f940edc2becd: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:15.739532 18721 tablet_bootstrap.cc:492] T e7995e0f8d084c659b0d702fda4acf66 P f56ca8f5d65d4597a2ac8651dae8d6b7: No bootstrap required, opened a new log
I20240503 14:12:15.739652 18721 ts_tablet_manager.cc:1367] T e7995e0f8d084c659b0d702fda4acf66 P f56ca8f5d65d4597a2ac8651dae8d6b7: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 14:12:15.739820 18719 raft_consensus.cc:348] T 340236565cbb4796ad23ed5d13c9f14e P e53c5f1c8f6f4e90ad01fb89fd81a05c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcda0acf1e7f40369d82f940edc2becd" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 46679 } } peers { permanent_uuid: "f56ca8f5d65d4597a2ac8651dae8d6b7" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39207 } } peers { permanent_uuid: "e53c5f1c8f6f4e90ad01fb89fd81a05c" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 46285 } }
I20240503 14:12:15.739992 18719 raft_consensus.cc:374] T 340236565cbb4796ad23ed5d13c9f14e P e53c5f1c8f6f4e90ad01fb89fd81a05c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:15.740118 18719 raft_consensus.cc:729] T 340236565cbb4796ad23ed5d13c9f14e P e53c5f1c8f6f4e90ad01fb89fd81a05c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e53c5f1c8f6f4e90ad01fb89fd81a05c, State: Initialized, Role: FOLLOWER
I20240503 14:12:15.740351 18719 consensus_queue.cc:260] T 340236565cbb4796ad23ed5d13c9f14e P e53c5f1c8f6f4e90ad01fb89fd81a05c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcda0acf1e7f40369d82f940edc2becd" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 46679 } } peers { permanent_uuid: "f56ca8f5d65d4597a2ac8651dae8d6b7" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39207 } } peers { permanent_uuid: "e53c5f1c8f6f4e90ad01fb89fd81a05c" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 46285 } }
I20240503 14:12:15.740428 18721 raft_consensus.cc:348] T e7995e0f8d084c659b0d702fda4acf66 P f56ca8f5d65d4597a2ac8651dae8d6b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f56ca8f5d65d4597a2ac8651dae8d6b7" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39207 } } peers { permanent_uuid: "dcda0acf1e7f40369d82f940edc2becd" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 46679 } } peers { permanent_uuid: "e53c5f1c8f6f4e90ad01fb89fd81a05c" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 46285 } }
I20240503 14:12:15.740622 18721 raft_consensus.cc:374] T e7995e0f8d084c659b0d702fda4acf66 P f56ca8f5d65d4597a2ac8651dae8d6b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:15.740732 18721 raft_consensus.cc:729] T e7995e0f8d084c659b0d702fda4acf66 P f56ca8f5d65d4597a2ac8651dae8d6b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f56ca8f5d65d4597a2ac8651dae8d6b7, State: Initialized, Role: FOLLOWER
I20240503 14:12:15.740948 18721 consensus_queue.cc:260] T e7995e0f8d084c659b0d702fda4acf66 P f56ca8f5d65d4597a2ac8651dae8d6b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f56ca8f5d65d4597a2ac8651dae8d6b7" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39207 } } peers { permanent_uuid: "dcda0acf1e7f40369d82f940edc2becd" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 46679 } } peers { permanent_uuid: "e53c5f1c8f6f4e90ad01fb89fd81a05c" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 46285 } }
I20240503 14:12:15.741293 18718 tablet_bootstrap.cc:492] T 340236565cbb4796ad23ed5d13c9f14e P dcda0acf1e7f40369d82f940edc2becd: No bootstrap required, opened a new log
I20240503 14:12:15.741468 18718 ts_tablet_manager.cc:1367] T 340236565cbb4796ad23ed5d13c9f14e P dcda0acf1e7f40369d82f940edc2becd: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:12:15.741793 18721 ts_tablet_manager.cc:1398] T e7995e0f8d084c659b0d702fda4acf66 P f56ca8f5d65d4597a2ac8651dae8d6b7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:15.742023 18721 tablet_bootstrap.cc:492] T 340236565cbb4796ad23ed5d13c9f14e P f56ca8f5d65d4597a2ac8651dae8d6b7: Bootstrap starting.
I20240503 14:12:15.742133 18718 raft_consensus.cc:348] T 340236565cbb4796ad23ed5d13c9f14e P dcda0acf1e7f40369d82f940edc2becd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcda0acf1e7f40369d82f940edc2becd" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 46679 } } peers { permanent_uuid: "f56ca8f5d65d4597a2ac8651dae8d6b7" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39207 } } peers { permanent_uuid: "e53c5f1c8f6f4e90ad01fb89fd81a05c" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 46285 } }
I20240503 14:12:15.742322 18718 raft_consensus.cc:374] T 340236565cbb4796ad23ed5d13c9f14e P dcda0acf1e7f40369d82f940edc2becd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:15.742390 18718 raft_consensus.cc:729] T 340236565cbb4796ad23ed5d13c9f14e P dcda0acf1e7f40369d82f940edc2becd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcda0acf1e7f40369d82f940edc2becd, State: Initialized, Role: FOLLOWER
I20240503 14:12:15.742625 18718 consensus_queue.cc:260] T 340236565cbb4796ad23ed5d13c9f14e P dcda0acf1e7f40369d82f940edc2becd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcda0acf1e7f40369d82f940edc2becd" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 46679 } } peers { permanent_uuid: "f56ca8f5d65d4597a2ac8651dae8d6b7" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39207 } } peers { permanent_uuid: "e53c5f1c8f6f4e90ad01fb89fd81a05c" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 46285 } }
I20240503 14:12:15.743113 18718 ts_tablet_manager.cc:1398] T 340236565cbb4796ad23ed5d13c9f14e P dcda0acf1e7f40369d82f940edc2becd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:15.743443 18721 tablet_bootstrap.cc:654] T 340236565cbb4796ad23ed5d13c9f14e P f56ca8f5d65d4597a2ac8651dae8d6b7: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:15.748606 18719 ts_tablet_manager.cc:1398] T 340236565cbb4796ad23ed5d13c9f14e P e53c5f1c8f6f4e90ad01fb89fd81a05c: Time spent starting tablet: real 0.009s user 0.009s sys 0.000s
I20240503 14:12:15.748893 18719 tablet_bootstrap.cc:492] T e7995e0f8d084c659b0d702fda4acf66 P e53c5f1c8f6f4e90ad01fb89fd81a05c: Bootstrap starting.
I20240503 14:12:15.749692 18721 tablet_bootstrap.cc:492] T 340236565cbb4796ad23ed5d13c9f14e P f56ca8f5d65d4597a2ac8651dae8d6b7: No bootstrap required, opened a new log
I20240503 14:12:15.749876 18721 ts_tablet_manager.cc:1367] T 340236565cbb4796ad23ed5d13c9f14e P f56ca8f5d65d4597a2ac8651dae8d6b7: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.005s
I20240503 14:12:15.750525 18719 tablet_bootstrap.cc:654] T e7995e0f8d084c659b0d702fda4acf66 P e53c5f1c8f6f4e90ad01fb89fd81a05c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:15.750684 18721 raft_consensus.cc:348] T 340236565cbb4796ad23ed5d13c9f14e P f56ca8f5d65d4597a2ac8651dae8d6b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcda0acf1e7f40369d82f940edc2becd" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 46679 } } peers { permanent_uuid: "f56ca8f5d65d4597a2ac8651dae8d6b7" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39207 } } peers { permanent_uuid: "e53c5f1c8f6f4e90ad01fb89fd81a05c" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 46285 } }
I20240503 14:12:15.750941 18721 raft_consensus.cc:374] T 340236565cbb4796ad23ed5d13c9f14e P f56ca8f5d65d4597a2ac8651dae8d6b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:15.751045 18721 raft_consensus.cc:729] T 340236565cbb4796ad23ed5d13c9f14e P f56ca8f5d65d4597a2ac8651dae8d6b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f56ca8f5d65d4597a2ac8651dae8d6b7, State: Initialized, Role: FOLLOWER
I20240503 14:12:15.751286 18721 consensus_queue.cc:260] T 340236565cbb4796ad23ed5d13c9f14e P f56ca8f5d65d4597a2ac8651dae8d6b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcda0acf1e7f40369d82f940edc2becd" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 46679 } } peers { permanent_uuid: "f56ca8f5d65d4597a2ac8651dae8d6b7" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39207 } } peers { permanent_uuid: "e53c5f1c8f6f4e90ad01fb89fd81a05c" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 46285 } }
I20240503 14:12:15.751741 18721 ts_tablet_manager.cc:1398] T 340236565cbb4796ad23ed5d13c9f14e P f56ca8f5d65d4597a2ac8651dae8d6b7: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:12:15.752795 18719 tablet_bootstrap.cc:492] T e7995e0f8d084c659b0d702fda4acf66 P e53c5f1c8f6f4e90ad01fb89fd81a05c: No bootstrap required, opened a new log
I20240503 14:12:15.752934 18719 ts_tablet_manager.cc:1367] T e7995e0f8d084c659b0d702fda4acf66 P e53c5f1c8f6f4e90ad01fb89fd81a05c: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:12:15.753486 18719 raft_consensus.cc:348] T e7995e0f8d084c659b0d702fda4acf66 P e53c5f1c8f6f4e90ad01fb89fd81a05c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f56ca8f5d65d4597a2ac8651dae8d6b7" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39207 } } peers { permanent_uuid: "dcda0acf1e7f40369d82f940edc2becd" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 46679 } } peers { permanent_uuid: "e53c5f1c8f6f4e90ad01fb89fd81a05c" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 46285 } }
I20240503 14:12:15.753648 18719 raft_consensus.cc:374] T e7995e0f8d084c659b0d702fda4acf66 P e53c5f1c8f6f4e90ad01fb89fd81a05c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:15.753760 18719 raft_consensus.cc:729] T e7995e0f8d084c659b0d702fda4acf66 P e53c5f1c8f6f4e90ad01fb89fd81a05c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e53c5f1c8f6f4e90ad01fb89fd81a05c, State: Initialized, Role: FOLLOWER
I20240503 14:12:15.754022 18719 consensus_queue.cc:260] T e7995e0f8d084c659b0d702fda4acf66 P e53c5f1c8f6f4e90ad01fb89fd81a05c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f56ca8f5d65d4597a2ac8651dae8d6b7" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39207 } } peers { permanent_uuid: "dcda0acf1e7f40369d82f940edc2becd" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 46679 } } peers { permanent_uuid: "e53c5f1c8f6f4e90ad01fb89fd81a05c" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 46285 } }
I20240503 14:12:15.754390 18719 ts_tablet_manager.cc:1398] T e7995e0f8d084c659b0d702fda4acf66 P e53c5f1c8f6f4e90ad01fb89fd81a05c: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 14:12:15.772928 18727 raft_consensus.cc:482] T 340236565cbb4796ad23ed5d13c9f14e P f56ca8f5d65d4597a2ac8651dae8d6b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:15.773079 18727 raft_consensus.cc:504] T 340236565cbb4796ad23ed5d13c9f14e P f56ca8f5d65d4597a2ac8651dae8d6b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcda0acf1e7f40369d82f940edc2becd" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 46679 } } peers { permanent_uuid: "f56ca8f5d65d4597a2ac8651dae8d6b7" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39207 } } peers { permanent_uuid: "e53c5f1c8f6f4e90ad01fb89fd81a05c" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 46285 } }
I20240503 14:12:15.773859 18727 leader_election.cc:288] T 340236565cbb4796ad23ed5d13c9f14e P f56ca8f5d65d4597a2ac8651dae8d6b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dcda0acf1e7f40369d82f940edc2becd (127.16.94.129:46679), e53c5f1c8f6f4e90ad01fb89fd81a05c (127.16.94.131:46285)
I20240503 14:12:15.780566 18522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "340236565cbb4796ad23ed5d13c9f14e" candidate_uuid: "f56ca8f5d65d4597a2ac8651dae8d6b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcda0acf1e7f40369d82f940edc2becd" is_pre_election: true
I20240503 14:12:15.780598 18672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "340236565cbb4796ad23ed5d13c9f14e" candidate_uuid: "f56ca8f5d65d4597a2ac8651dae8d6b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e53c5f1c8f6f4e90ad01fb89fd81a05c" is_pre_election: true
I20240503 14:12:15.780835 18672 raft_consensus.cc:2454] T 340236565cbb4796ad23ed5d13c9f14e P e53c5f1c8f6f4e90ad01fb89fd81a05c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f56ca8f5d65d4597a2ac8651dae8d6b7 in term 0.
I20240503 14:12:15.780833 18522 raft_consensus.cc:2454] T 340236565cbb4796ad23ed5d13c9f14e P dcda0acf1e7f40369d82f940edc2becd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f56ca8f5d65d4597a2ac8651dae8d6b7 in term 0.
I20240503 14:12:15.781265 18564 leader_election.cc:302] T 340236565cbb4796ad23ed5d13c9f14e P f56ca8f5d65d4597a2ac8651dae8d6b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e53c5f1c8f6f4e90ad01fb89fd81a05c, f56ca8f5d65d4597a2ac8651dae8d6b7; no voters:
I20240503 14:12:15.781488 18727 raft_consensus.cc:2781] T 340236565cbb4796ad23ed5d13c9f14e P f56ca8f5d65d4597a2ac8651dae8d6b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:15.781597 18727 raft_consensus.cc:482] T 340236565cbb4796ad23ed5d13c9f14e P f56ca8f5d65d4597a2ac8651dae8d6b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:15.781706 18727 raft_consensus.cc:3037] T 340236565cbb4796ad23ed5d13c9f14e P f56ca8f5d65d4597a2ac8651dae8d6b7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:15.783393 18727 raft_consensus.cc:504] T 340236565cbb4796ad23ed5d13c9f14e P f56ca8f5d65d4597a2ac8651dae8d6b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcda0acf1e7f40369d82f940edc2becd" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 46679 } } peers { permanent_uuid: "f56ca8f5d65d4597a2ac8651dae8d6b7" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39207 } } peers { permanent_uuid: "e53c5f1c8f6f4e90ad01fb89fd81a05c" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 46285 } }
I20240503 14:12:15.783886 18727 leader_election.cc:288] T 340236565cbb4796ad23ed5d13c9f14e P f56ca8f5d65d4597a2ac8651dae8d6b7 [CANDIDATE]: Term 1 election: Requested vote from peers dcda0acf1e7f40369d82f940edc2becd (127.16.94.129:46679), e53c5f1c8f6f4e90ad01fb89fd81a05c (127.16.94.131:46285)
I20240503 14:12:15.784214 18522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "340236565cbb4796ad23ed5d13c9f14e" candidate_uuid: "f56ca8f5d65d4597a2ac8651dae8d6b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcda0acf1e7f40369d82f940edc2becd"
I20240503 14:12:15.784297 18672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "340236565cbb4796ad23ed5d13c9f14e" candidate_uuid: "f56ca8f5d65d4597a2ac8651dae8d6b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e53c5f1c8f6f4e90ad01fb89fd81a05c"
I20240503 14:12:15.784379 18522 raft_consensus.cc:3037] T 340236565cbb4796ad23ed5d13c9f14e P dcda0acf1e7f40369d82f940edc2becd [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:15.784461 18672 raft_consensus.cc:3037] T 340236565cbb4796ad23ed5d13c9f14e P e53c5f1c8f6f4e90ad01fb89fd81a05c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:15.786278 18672 raft_consensus.cc:2454] T 340236565cbb4796ad23ed5d13c9f14e P e53c5f1c8f6f4e90ad01fb89fd81a05c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f56ca8f5d65d4597a2ac8651dae8d6b7 in term 1.
I20240503 14:12:15.786300 18522 raft_consensus.cc:2454] T 340236565cbb4796ad23ed5d13c9f14e P dcda0acf1e7f40369d82f940edc2becd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f56ca8f5d65d4597a2ac8651dae8d6b7 in term 1.
I20240503 14:12:15.786581 18564 leader_election.cc:302] T 340236565cbb4796ad23ed5d13c9f14e P f56ca8f5d65d4597a2ac8651dae8d6b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e53c5f1c8f6f4e90ad01fb89fd81a05c, f56ca8f5d65d4597a2ac8651dae8d6b7; no voters:
I20240503 14:12:15.786808 18727 raft_consensus.cc:2781] T 340236565cbb4796ad23ed5d13c9f14e P f56ca8f5d65d4597a2ac8651dae8d6b7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:15.787147 18727 raft_consensus.cc:686] T 340236565cbb4796ad23ed5d13c9f14e P f56ca8f5d65d4597a2ac8651dae8d6b7 [term 1 LEADER]: Becoming Leader. State: Replica: f56ca8f5d65d4597a2ac8651dae8d6b7, State: Running, Role: LEADER
I20240503 14:12:15.787397 18727 consensus_queue.cc:237] T 340236565cbb4796ad23ed5d13c9f14e P f56ca8f5d65d4597a2ac8651dae8d6b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcda0acf1e7f40369d82f940edc2becd" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 46679 } } peers { permanent_uuid: "f56ca8f5d65d4597a2ac8651dae8d6b7" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39207 } } peers { permanent_uuid: "e53c5f1c8f6f4e90ad01fb89fd81a05c" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 46285 } }
I20240503 14:12:15.789796 18420 catalog_manager.cc:5484] T 340236565cbb4796ad23ed5d13c9f14e P f56ca8f5d65d4597a2ac8651dae8d6b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to f56ca8f5d65d4597a2ac8651dae8d6b7 (127.16.94.130). New cstate: current_term: 1 leader_uuid: "f56ca8f5d65d4597a2ac8651dae8d6b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcda0acf1e7f40369d82f940edc2becd" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 46679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f56ca8f5d65d4597a2ac8651dae8d6b7" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e53c5f1c8f6f4e90ad01fb89fd81a05c" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 46285 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:15.821166 18727 raft_consensus.cc:482] T e7995e0f8d084c659b0d702fda4acf66 P f56ca8f5d65d4597a2ac8651dae8d6b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:15.821373 18727 raft_consensus.cc:504] T e7995e0f8d084c659b0d702fda4acf66 P f56ca8f5d65d4597a2ac8651dae8d6b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f56ca8f5d65d4597a2ac8651dae8d6b7" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39207 } } peers { permanent_uuid: "dcda0acf1e7f40369d82f940edc2becd" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 46679 } } peers { permanent_uuid: "e53c5f1c8f6f4e90ad01fb89fd81a05c" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 46285 } }
I20240503 14:12:15.821913 18727 leader_election.cc:288] T e7995e0f8d084c659b0d702fda4acf66 P f56ca8f5d65d4597a2ac8651dae8d6b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dcda0acf1e7f40369d82f940edc2becd (127.16.94.129:46679), e53c5f1c8f6f4e90ad01fb89fd81a05c (127.16.94.131:46285)
I20240503 14:12:15.822341 18522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7995e0f8d084c659b0d702fda4acf66" candidate_uuid: "f56ca8f5d65d4597a2ac8651dae8d6b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcda0acf1e7f40369d82f940edc2becd" is_pre_election: true
I20240503 14:12:15.822407 18672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7995e0f8d084c659b0d702fda4acf66" candidate_uuid: "f56ca8f5d65d4597a2ac8651dae8d6b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e53c5f1c8f6f4e90ad01fb89fd81a05c" is_pre_election: true
I20240503 14:12:15.822547 18522 raft_consensus.cc:2454] T e7995e0f8d084c659b0d702fda4acf66 P dcda0acf1e7f40369d82f940edc2becd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f56ca8f5d65d4597a2ac8651dae8d6b7 in term 0.
I20240503 14:12:15.822610 18672 raft_consensus.cc:2454] T e7995e0f8d084c659b0d702fda4acf66 P e53c5f1c8f6f4e90ad01fb89fd81a05c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f56ca8f5d65d4597a2ac8651dae8d6b7 in term 0.
I20240503 14:12:15.822902 18564 leader_election.cc:302] T e7995e0f8d084c659b0d702fda4acf66 P f56ca8f5d65d4597a2ac8651dae8d6b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dcda0acf1e7f40369d82f940edc2becd, f56ca8f5d65d4597a2ac8651dae8d6b7; no voters:
I20240503 14:12:15.823146 18727 raft_consensus.cc:2781] T e7995e0f8d084c659b0d702fda4acf66 P f56ca8f5d65d4597a2ac8651dae8d6b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:15.823232 18727 raft_consensus.cc:482] T e7995e0f8d084c659b0d702fda4acf66 P f56ca8f5d65d4597a2ac8651dae8d6b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:15.823320 18727 raft_consensus.cc:3037] T e7995e0f8d084c659b0d702fda4acf66 P f56ca8f5d65d4597a2ac8651dae8d6b7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:15.824970 18727 raft_consensus.cc:504] T e7995e0f8d084c659b0d702fda4acf66 P f56ca8f5d65d4597a2ac8651dae8d6b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f56ca8f5d65d4597a2ac8651dae8d6b7" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39207 } } peers { permanent_uuid: "dcda0acf1e7f40369d82f940edc2becd" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 46679 } } peers { permanent_uuid: "e53c5f1c8f6f4e90ad01fb89fd81a05c" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 46285 } }
I20240503 14:12:15.825443 18727 leader_election.cc:288] T e7995e0f8d084c659b0d702fda4acf66 P f56ca8f5d65d4597a2ac8651dae8d6b7 [CANDIDATE]: Term 1 election: Requested vote from peers dcda0acf1e7f40369d82f940edc2becd (127.16.94.129:46679), e53c5f1c8f6f4e90ad01fb89fd81a05c (127.16.94.131:46285)
I20240503 14:12:15.825819 18522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7995e0f8d084c659b0d702fda4acf66" candidate_uuid: "f56ca8f5d65d4597a2ac8651dae8d6b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcda0acf1e7f40369d82f940edc2becd"
I20240503 14:12:15.825852 18672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7995e0f8d084c659b0d702fda4acf66" candidate_uuid: "f56ca8f5d65d4597a2ac8651dae8d6b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e53c5f1c8f6f4e90ad01fb89fd81a05c"
I20240503 14:12:15.825978 18522 raft_consensus.cc:3037] T e7995e0f8d084c659b0d702fda4acf66 P dcda0acf1e7f40369d82f940edc2becd [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:15.826014 18672 raft_consensus.cc:3037] T e7995e0f8d084c659b0d702fda4acf66 P e53c5f1c8f6f4e90ad01fb89fd81a05c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:15.827731 18672 raft_consensus.cc:2454] T e7995e0f8d084c659b0d702fda4acf66 P e53c5f1c8f6f4e90ad01fb89fd81a05c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f56ca8f5d65d4597a2ac8651dae8d6b7 in term 1.
I20240503 14:12:15.827728 18522 raft_consensus.cc:2454] T e7995e0f8d084c659b0d702fda4acf66 P dcda0acf1e7f40369d82f940edc2becd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f56ca8f5d65d4597a2ac8651dae8d6b7 in term 1.
I20240503 14:12:15.828085 18564 leader_election.cc:302] T e7995e0f8d084c659b0d702fda4acf66 P f56ca8f5d65d4597a2ac8651dae8d6b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e53c5f1c8f6f4e90ad01fb89fd81a05c, f56ca8f5d65d4597a2ac8651dae8d6b7; no voters:
I20240503 14:12:15.828313 18727 raft_consensus.cc:2781] T e7995e0f8d084c659b0d702fda4acf66 P f56ca8f5d65d4597a2ac8651dae8d6b7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:15.828475 18727 raft_consensus.cc:686] T e7995e0f8d084c659b0d702fda4acf66 P f56ca8f5d65d4597a2ac8651dae8d6b7 [term 1 LEADER]: Becoming Leader. State: Replica: f56ca8f5d65d4597a2ac8651dae8d6b7, State: Running, Role: LEADER
I20240503 14:12:15.828683 18727 consensus_queue.cc:237] T e7995e0f8d084c659b0d702fda4acf66 P f56ca8f5d65d4597a2ac8651dae8d6b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f56ca8f5d65d4597a2ac8651dae8d6b7" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39207 } } peers { permanent_uuid: "dcda0acf1e7f40369d82f940edc2becd" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 46679 } } peers { permanent_uuid: "e53c5f1c8f6f4e90ad01fb89fd81a05c" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 46285 } }
I20240503 14:12:15.830771 18420 catalog_manager.cc:5484] T e7995e0f8d084c659b0d702fda4acf66 P f56ca8f5d65d4597a2ac8651dae8d6b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to f56ca8f5d65d4597a2ac8651dae8d6b7 (127.16.94.130). New cstate: current_term: 1 leader_uuid: "f56ca8f5d65d4597a2ac8651dae8d6b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f56ca8f5d65d4597a2ac8651dae8d6b7" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dcda0acf1e7f40369d82f940edc2becd" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 46679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e53c5f1c8f6f4e90ad01fb89fd81a05c" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 46285 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:16.259243 18727 consensus_queue.cc:1035] T e7995e0f8d084c659b0d702fda4acf66 P f56ca8f5d65d4597a2ac8651dae8d6b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e53c5f1c8f6f4e90ad01fb89fd81a05c" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 46285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:16.264456 18732 consensus_queue.cc:1035] T e7995e0f8d084c659b0d702fda4acf66 P f56ca8f5d65d4597a2ac8651dae8d6b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcda0acf1e7f40369d82f940edc2becd" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 46679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:16.318770 18738 consensus_queue.cc:1035] T 340236565cbb4796ad23ed5d13c9f14e P f56ca8f5d65d4597a2ac8651dae8d6b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e53c5f1c8f6f4e90ad01fb89fd81a05c" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 46285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:16.322650 18744 consensus_queue.cc:1035] T 340236565cbb4796ad23ed5d13c9f14e P f56ca8f5d65d4597a2ac8651dae8d6b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcda0acf1e7f40369d82f940edc2becd" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 46679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:16.463665 18474 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:12:16.470060 18598 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "340236565cbb4796ad23ed5d13c9f14e"
dest_uuid: "f56ca8f5d65d4597a2ac8651dae8d6b7"
mode: GRACEFUL
new_leader_uuid: "dcda0acf1e7f40369d82f940edc2becd"
from {username='slave'} at 127.0.0.1:47752
I20240503 14:12:16.470285 18598 raft_consensus.cc:595] T 340236565cbb4796ad23ed5d13c9f14e P f56ca8f5d65d4597a2ac8651dae8d6b7 [term 1 LEADER]: Received request to transfer leadership to TS dcda0acf1e7f40369d82f940edc2becd
I20240503 14:12:16.470801 18474 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240503 14:12:16.470921 18474 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:12:16.908047 18744 raft_consensus.cc:979] T 340236565cbb4796ad23ed5d13c9f14e P f56ca8f5d65d4597a2ac8651dae8d6b7: : Instructing follower dcda0acf1e7f40369d82f940edc2becd to start an election
I20240503 14:12:16.908210 18744 raft_consensus.cc:1067] T 340236565cbb4796ad23ed5d13c9f14e P f56ca8f5d65d4597a2ac8651dae8d6b7 [term 1 LEADER]: Signalling peer dcda0acf1e7f40369d82f940edc2becd to start an election
I20240503 14:12:16.909036 18522 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "340236565cbb4796ad23ed5d13c9f14e"
dest_uuid: "dcda0acf1e7f40369d82f940edc2becd"
from {username='slave'} at 127.0.0.1:57042
I20240503 14:12:16.909245 18522 raft_consensus.cc:482] T 340236565cbb4796ad23ed5d13c9f14e P dcda0acf1e7f40369d82f940edc2becd [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:12:16.909348 18522 raft_consensus.cc:3037] T 340236565cbb4796ad23ed5d13c9f14e P dcda0acf1e7f40369d82f940edc2becd [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:12:16.911203 18522 raft_consensus.cc:504] T 340236565cbb4796ad23ed5d13c9f14e P dcda0acf1e7f40369d82f940edc2becd [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcda0acf1e7f40369d82f940edc2becd" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 46679 } } peers { permanent_uuid: "f56ca8f5d65d4597a2ac8651dae8d6b7" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39207 } } peers { permanent_uuid: "e53c5f1c8f6f4e90ad01fb89fd81a05c" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 46285 } }
I20240503 14:12:16.912001 18522 leader_election.cc:288] T 340236565cbb4796ad23ed5d13c9f14e P dcda0acf1e7f40369d82f940edc2becd [CANDIDATE]: Term 2 election: Requested vote from peers f56ca8f5d65d4597a2ac8651dae8d6b7 (127.16.94.130:39207), e53c5f1c8f6f4e90ad01fb89fd81a05c (127.16.94.131:46285)
I20240503 14:12:16.918347 18598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "340236565cbb4796ad23ed5d13c9f14e" candidate_uuid: "dcda0acf1e7f40369d82f940edc2becd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f56ca8f5d65d4597a2ac8651dae8d6b7"
I20240503 14:12:16.918373 18672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "340236565cbb4796ad23ed5d13c9f14e" candidate_uuid: "dcda0acf1e7f40369d82f940edc2becd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e53c5f1c8f6f4e90ad01fb89fd81a05c"
I20240503 14:12:16.918537 18598 raft_consensus.cc:3032] T 340236565cbb4796ad23ed5d13c9f14e P f56ca8f5d65d4597a2ac8651dae8d6b7 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:12:16.918584 18672 raft_consensus.cc:3037] T 340236565cbb4796ad23ed5d13c9f14e P e53c5f1c8f6f4e90ad01fb89fd81a05c [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:12:16.918653 18598 raft_consensus.cc:729] T 340236565cbb4796ad23ed5d13c9f14e P f56ca8f5d65d4597a2ac8651dae8d6b7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f56ca8f5d65d4597a2ac8651dae8d6b7, State: Running, Role: LEADER
I20240503 14:12:16.918946 18598 consensus_queue.cc:260] T 340236565cbb4796ad23ed5d13c9f14e P f56ca8f5d65d4597a2ac8651dae8d6b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcda0acf1e7f40369d82f940edc2becd" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 46679 } } peers { permanent_uuid: "f56ca8f5d65d4597a2ac8651dae8d6b7" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39207 } } peers { permanent_uuid: "e53c5f1c8f6f4e90ad01fb89fd81a05c" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 46285 } }
I20240503 14:12:16.919265 18598 raft_consensus.cc:3037] T 340236565cbb4796ad23ed5d13c9f14e P f56ca8f5d65d4597a2ac8651dae8d6b7 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:12:16.920415 18672 raft_consensus.cc:2454] T 340236565cbb4796ad23ed5d13c9f14e P e53c5f1c8f6f4e90ad01fb89fd81a05c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcda0acf1e7f40369d82f940edc2becd in term 2.
I20240503 14:12:16.920801 18488 leader_election.cc:302] T 340236565cbb4796ad23ed5d13c9f14e P dcda0acf1e7f40369d82f940edc2becd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dcda0acf1e7f40369d82f940edc2becd, e53c5f1c8f6f4e90ad01fb89fd81a05c; no voters:
I20240503 14:12:16.921063 18598 raft_consensus.cc:2454] T 340236565cbb4796ad23ed5d13c9f14e P f56ca8f5d65d4597a2ac8651dae8d6b7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcda0acf1e7f40369d82f940edc2becd in term 2.
I20240503 14:12:16.921322 18751 raft_consensus.cc:2781] T 340236565cbb4796ad23ed5d13c9f14e P dcda0acf1e7f40369d82f940edc2becd [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:12:16.921749 18751 raft_consensus.cc:686] T 340236565cbb4796ad23ed5d13c9f14e P dcda0acf1e7f40369d82f940edc2becd [term 2 LEADER]: Becoming Leader. State: Replica: dcda0acf1e7f40369d82f940edc2becd, State: Running, Role: LEADER
I20240503 14:12:16.921988 18751 consensus_queue.cc:237] T 340236565cbb4796ad23ed5d13c9f14e P dcda0acf1e7f40369d82f940edc2becd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcda0acf1e7f40369d82f940edc2becd" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 46679 } } peers { permanent_uuid: "f56ca8f5d65d4597a2ac8651dae8d6b7" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39207 } } peers { permanent_uuid: "e53c5f1c8f6f4e90ad01fb89fd81a05c" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 46285 } }
I20240503 14:12:16.923704 18419 catalog_manager.cc:5484] T 340236565cbb4796ad23ed5d13c9f14e P dcda0acf1e7f40369d82f940edc2becd reported cstate change: term changed from 1 to 2, leader changed from f56ca8f5d65d4597a2ac8651dae8d6b7 (127.16.94.130) to dcda0acf1e7f40369d82f940edc2becd (127.16.94.129). New cstate: current_term: 2 leader_uuid: "dcda0acf1e7f40369d82f940edc2becd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcda0acf1e7f40369d82f940edc2becd" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 46679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f56ca8f5d65d4597a2ac8651dae8d6b7" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e53c5f1c8f6f4e90ad01fb89fd81a05c" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 46285 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:17.440320 18598 raft_consensus.cc:1261] T 340236565cbb4796ad23ed5d13c9f14e P f56ca8f5d65d4597a2ac8651dae8d6b7 [term 2 FOLLOWER]: Refusing update from remote peer dcda0acf1e7f40369d82f940edc2becd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 14:12:17.440845 18757 consensus_queue.cc:1035] T 340236565cbb4796ad23ed5d13c9f14e P dcda0acf1e7f40369d82f940edc2becd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f56ca8f5d65d4597a2ac8651dae8d6b7" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:12:17.444512 18672 raft_consensus.cc:1261] T 340236565cbb4796ad23ed5d13c9f14e P e53c5f1c8f6f4e90ad01fb89fd81a05c [term 2 FOLLOWER]: Refusing update from remote peer dcda0acf1e7f40369d82f940edc2becd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 14:12:17.445045 18759 consensus_queue.cc:1035] T 340236565cbb4796ad23ed5d13c9f14e P dcda0acf1e7f40369d82f940edc2becd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e53c5f1c8f6f4e90ad01fb89fd81a05c" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 46285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:12:18.471220 18474 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:12:18.471766 18474 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:12:18.471884 18474 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:12:20.472286 18474 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:12:20.472805 18474 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:12:20.472944 18474 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:12:22.473258 18474 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:12:22.473980 18474 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:12:22.474118 18474 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:12:24.474439 18474 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:12:24.475088 18474 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:12:24.475212 18474 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:12:26.475554 18474 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:12:26.476157 18474 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:12:26.476276 18474 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:12:26.887943 16762 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:26.891122 18770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:26.891925 18771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:26.892453 18773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:26.893555 16762 server_base.cc:1034] running on GCE node
I20240503 14:12:26.894886 16762 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:12:26.894955 16762 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:12:26.895026 16762 hybrid_clock.cc:648] HybridClock initialized: now 1714745546894993 us; error 0 us; skew 500 ppm
I20240503 14:12:26.895181 16762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:26.901777 16762 webserver.cc:460] Webserver started at http://127.16.94.132:36533/ using document root <none> and password file <none>
I20240503 14:12:26.902006 16762 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:26.902108 16762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:26.902251 16762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:26.902878 16762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714745515241994-16762-0/minicluster-data/ts-3-root/instance:
uuid: "64ed0d0908264ff0813baed49c2636e1"
format_stamp: "Formatted at 2024-05-03 14:12:26 on dist-test-slave-v27g"
I20240503 14:12:26.905228 16762 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20240503 14:12:26.906796 18778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:26.907191 16762 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240503 14:12:26.907294 16762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714745515241994-16762-0/minicluster-data/ts-3-root
uuid: "64ed0d0908264ff0813baed49c2636e1"
format_stamp: "Formatted at 2024-05-03 14:12:26 on dist-test-slave-v27g"
I20240503 14:12:26.907425 16762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714745515241994-16762-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714745515241994-16762-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714745515241994-16762-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:26.914430 16762 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:26.914860 16762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:26.915357 16762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:26.916112 16762 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:26.916172 16762 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:26.916290 16762 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:26.916347 16762 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:26.930100 16762 rpc_server.cc:307] RPC server started. Bound to: 127.16.94.132:42903
I20240503 14:12:26.930123 18840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.94.132:42903 every 8 connection(s)
I20240503 14:12:26.937011 18841 heartbeater.cc:347] Connected to a master server at 127.16.94.190:44641
I20240503 14:12:26.937155 18841 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:26.937644 18841 heartbeater.cc:511] Master 127.16.94.190:44641 requested a full tablet report, sending...
I20240503 14:12:26.938422 18420 ts_manager.cc:194] Registered new tserver with Master: 64ed0d0908264ff0813baed49c2636e1 (127.16.94.132:42903)
I20240503 14:12:26.939103 18420 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36244
I20240503 14:12:27.484349 18522 consensus_queue.cc:237] T 340236565cbb4796ad23ed5d13c9f14e P dcda0acf1e7f40369d82f940edc2becd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dcda0acf1e7f40369d82f940edc2becd" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 46679 } } peers { permanent_uuid: "f56ca8f5d65d4597a2ac8651dae8d6b7" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39207 } } peers { permanent_uuid: "e53c5f1c8f6f4e90ad01fb89fd81a05c" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 46285 } attrs { replace: true } } peers { permanent_uuid: "64ed0d0908264ff0813baed49c2636e1" member_type: NON_VOTER last_known_addr { host: "127.16.94.132" port: 42903 } attrs { promote: true } }
I20240503 14:12:27.486703 18598 raft_consensus.cc:1261] T 340236565cbb4796ad23ed5d13c9f14e P f56ca8f5d65d4597a2ac8651dae8d6b7 [term 2 FOLLOWER]: Refusing update from remote peer dcda0acf1e7f40369d82f940edc2becd: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 14:12:27.487041 18672 raft_consensus.cc:1261] T 340236565cbb4796ad23ed5d13c9f14e P e53c5f1c8f6f4e90ad01fb89fd81a05c [term 2 FOLLOWER]: Refusing update from remote peer dcda0acf1e7f40369d82f940edc2becd: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 14:12:27.487344 18849 consensus_queue.cc:1035] T 340236565cbb4796ad23ed5d13c9f14e P dcda0acf1e7f40369d82f940edc2becd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f56ca8f5d65d4597a2ac8651dae8d6b7" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:12:27.487643 18769 consensus_queue.cc:1035] T 340236565cbb4796ad23ed5d13c9f14e P dcda0acf1e7f40369d82f940edc2becd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e53c5f1c8f6f4e90ad01fb89fd81a05c" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 46285 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:12:27.490944 18769 raft_consensus.cc:2932] T 340236565cbb4796ad23ed5d13c9f14e P dcda0acf1e7f40369d82f940edc2becd [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 64ed0d0908264ff0813baed49c2636e1 (127.16.94.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dcda0acf1e7f40369d82f940edc2becd" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 46679 } } peers { permanent_uuid: "f56ca8f5d65d4597a2ac8651dae8d6b7" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39207 } } peers { permanent_uuid: "e53c5f1c8f6f4e90ad01fb89fd81a05c" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 46285 } attrs { replace: true } } peers { permanent_uuid: "64ed0d0908264ff0813baed49c2636e1" member_type: NON_VOTER last_known_addr { host: "127.16.94.132" port: 42903 } attrs { promote: true } } }
I20240503 14:12:27.491596 18598 raft_consensus.cc:2932] T 340236565cbb4796ad23ed5d13c9f14e P f56ca8f5d65d4597a2ac8651dae8d6b7 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 64ed0d0908264ff0813baed49c2636e1 (127.16.94.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dcda0acf1e7f40369d82f940edc2becd" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 46679 } } peers { permanent_uuid: "f56ca8f5d65d4597a2ac8651dae8d6b7" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39207 } } peers { permanent_uuid: "e53c5f1c8f6f4e90ad01fb89fd81a05c" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 46285 } attrs { replace: true } } peers { permanent_uuid: "64ed0d0908264ff0813baed49c2636e1" member_type: NON_VOTER last_known_addr { host: "127.16.94.132" port: 42903 } attrs { promote: true } } }
I20240503 14:12:27.493407 18672 raft_consensus.cc:2932] T 340236565cbb4796ad23ed5d13c9f14e P e53c5f1c8f6f4e90ad01fb89fd81a05c [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 64ed0d0908264ff0813baed49c2636e1 (127.16.94.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dcda0acf1e7f40369d82f940edc2becd" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 46679 } } peers { permanent_uuid: "f56ca8f5d65d4597a2ac8651dae8d6b7" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39207 } } peers { permanent_uuid: "e53c5f1c8f6f4e90ad01fb89fd81a05c" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 46285 } attrs { replace: true } } peers { permanent_uuid: "64ed0d0908264ff0813baed49c2636e1" member_type: NON_VOTER last_known_addr { host: "127.16.94.132" port: 42903 } attrs { promote: true } } }
I20240503 14:12:27.496412 18419 catalog_manager.cc:5484] T 340236565cbb4796ad23ed5d13c9f14e P f56ca8f5d65d4597a2ac8651dae8d6b7 reported cstate change: config changed from index -1 to 3, NON_VOTER 64ed0d0908264ff0813baed49c2636e1 (127.16.94.132) added. New cstate: current_term: 2 leader_uuid: "dcda0acf1e7f40369d82f940edc2becd" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dcda0acf1e7f40369d82f940edc2becd" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 46679 } } peers { permanent_uuid: "f56ca8f5d65d4597a2ac8651dae8d6b7" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39207 } } peers { permanent_uuid: "e53c5f1c8f6f4e90ad01fb89fd81a05c" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 46285 } attrs { replace: true } } peers { permanent_uuid: "64ed0d0908264ff0813baed49c2636e1" member_type: NON_VOTER last_known_addr { host: "127.16.94.132" port: 42903 } attrs { promote: true } } }
W20240503 14:12:27.496805 18488 consensus_peers.cc:485] T 340236565cbb4796ad23ed5d13c9f14e P dcda0acf1e7f40369d82f940edc2becd -> Peer 64ed0d0908264ff0813baed49c2636e1 (127.16.94.132:42903): Couldn't send request to peer 64ed0d0908264ff0813baed49c2636e1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 340236565cbb4796ad23ed5d13c9f14e. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:27.914327 18859 ts_tablet_manager.cc:902] T 340236565cbb4796ad23ed5d13c9f14e P 64ed0d0908264ff0813baed49c2636e1: Initiating tablet copy from peer dcda0acf1e7f40369d82f940edc2becd (127.16.94.129:46679)
I20240503 14:12:27.915992 18859 tablet_copy_client.cc:308] T 340236565cbb4796ad23ed5d13c9f14e P 64ed0d0908264ff0813baed49c2636e1: tablet copy: Beginning tablet copy session from remote peer at address 127.16.94.129:46679
I20240503 14:12:27.921736 18532 tablet_copy_service.cc:140] P dcda0acf1e7f40369d82f940edc2becd: Received BeginTabletCopySession request for tablet 340236565cbb4796ad23ed5d13c9f14e from peer 64ed0d0908264ff0813baed49c2636e1 ({username='slave'} at 127.0.0.1:49258)
I20240503 14:12:27.921887 18532 tablet_copy_service.cc:161] P dcda0acf1e7f40369d82f940edc2becd: Beginning new tablet copy session on tablet 340236565cbb4796ad23ed5d13c9f14e from peer 64ed0d0908264ff0813baed49c2636e1 at {username='slave'} at 127.0.0.1:49258: session id = 64ed0d0908264ff0813baed49c2636e1-340236565cbb4796ad23ed5d13c9f14e
I20240503 14:12:27.924165 18532 tablet_copy_source_session.cc:205] T 340236565cbb4796ad23ed5d13c9f14e P dcda0acf1e7f40369d82f940edc2becd: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:12:27.925287 18859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 340236565cbb4796ad23ed5d13c9f14e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:27.931038 18859 tablet_copy_client.cc:783] T 340236565cbb4796ad23ed5d13c9f14e P 64ed0d0908264ff0813baed49c2636e1: tablet copy: Starting download of 0 data blocks...
I20240503 14:12:27.931362 18859 tablet_copy_client.cc:647] T 340236565cbb4796ad23ed5d13c9f14e P 64ed0d0908264ff0813baed49c2636e1: tablet copy: Starting download of 1 WAL segments...
I20240503 14:12:27.932775 18859 tablet_copy_client.cc:523] T 340236565cbb4796ad23ed5d13c9f14e P 64ed0d0908264ff0813baed49c2636e1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:12:27.936275 18859 tablet_bootstrap.cc:492] T 340236565cbb4796ad23ed5d13c9f14e P 64ed0d0908264ff0813baed49c2636e1: Bootstrap starting.
I20240503 14:12:27.940109 18841 heartbeater.cc:503] Master 127.16.94.190:44641 was elected leader, sending a full tablet report...
I20240503 14:12:27.943333 18859 tablet_bootstrap.cc:492] T 340236565cbb4796ad23ed5d13c9f14e P 64ed0d0908264ff0813baed49c2636e1: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:12:27.943636 18859 tablet_bootstrap.cc:492] T 340236565cbb4796ad23ed5d13c9f14e P 64ed0d0908264ff0813baed49c2636e1: Bootstrap complete.
I20240503 14:12:27.943847 18859 ts_tablet_manager.cc:1367] T 340236565cbb4796ad23ed5d13c9f14e P 64ed0d0908264ff0813baed49c2636e1: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.001s
I20240503 14:12:27.944393 18859 raft_consensus.cc:348] T 340236565cbb4796ad23ed5d13c9f14e P 64ed0d0908264ff0813baed49c2636e1 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dcda0acf1e7f40369d82f940edc2becd" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 46679 } } peers { permanent_uuid: "f56ca8f5d65d4597a2ac8651dae8d6b7" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39207 } } peers { permanent_uuid: "e53c5f1c8f6f4e90ad01fb89fd81a05c" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 46285 } attrs { replace: true } } peers { permanent_uuid: "64ed0d0908264ff0813baed49c2636e1" member_type: NON_VOTER last_known_addr { host: "127.16.94.132" port: 42903 } attrs { promote: true } }
I20240503 14:12:27.944573 18859 raft_consensus.cc:729] T 340236565cbb4796ad23ed5d13c9f14e P 64ed0d0908264ff0813baed49c2636e1 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 64ed0d0908264ff0813baed49c2636e1, State: Initialized, Role: LEARNER
I20240503 14:12:27.944761 18859 consensus_queue.cc:260] T 340236565cbb4796ad23ed5d13c9f14e P 64ed0d0908264ff0813baed49c2636e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dcda0acf1e7f40369d82f940edc2becd" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 46679 } } peers { permanent_uuid: "f56ca8f5d65d4597a2ac8651dae8d6b7" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39207 } } peers { permanent_uuid: "e53c5f1c8f6f4e90ad01fb89fd81a05c" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 46285 } attrs { replace: true } } peers { permanent_uuid: "64ed0d0908264ff0813baed49c2636e1" member_type: NON_VOTER last_known_addr { host: "127.16.94.132" port: 42903 } attrs { promote: true } }
I20240503 14:12:27.945501 18859 ts_tablet_manager.cc:1398] T 340236565cbb4796ad23ed5d13c9f14e P 64ed0d0908264ff0813baed49c2636e1: Time spent starting tablet: real 0.002s user 0.003s sys 0.001s
I20240503 14:12:27.946336 18532 tablet_copy_service.cc:342] P dcda0acf1e7f40369d82f940edc2becd: Request end of tablet copy session 64ed0d0908264ff0813baed49c2636e1-340236565cbb4796ad23ed5d13c9f14e received from {username='slave'} at 127.0.0.1:49258
I20240503 14:12:27.946487 18532 tablet_copy_service.cc:433] P dcda0acf1e7f40369d82f940edc2becd: ending tablet copy session 64ed0d0908264ff0813baed49c2636e1-340236565cbb4796ad23ed5d13c9f14e on tablet 340236565cbb4796ad23ed5d13c9f14e with peer 64ed0d0908264ff0813baed49c2636e1
I20240503 14:12:28.441037 18816 raft_consensus.cc:1203] T 340236565cbb4796ad23ed5d13c9f14e P 64ed0d0908264ff0813baed49c2636e1 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20240503 14:12:28.476572 18474 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:12:28.477212 18474 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:12:28.477324 18474 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:12:29.031158 18868 raft_consensus.cc:1050] T 340236565cbb4796ad23ed5d13c9f14e P dcda0acf1e7f40369d82f940edc2becd: attempting to promote NON_VOTER 64ed0d0908264ff0813baed49c2636e1 to VOTER
I20240503 14:12:29.032002 18868 consensus_queue.cc:237] T 340236565cbb4796ad23ed5d13c9f14e P dcda0acf1e7f40369d82f940edc2becd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dcda0acf1e7f40369d82f940edc2becd" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 46679 } } peers { permanent_uuid: "f56ca8f5d65d4597a2ac8651dae8d6b7" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39207 } } peers { permanent_uuid: "e53c5f1c8f6f4e90ad01fb89fd81a05c" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 46285 } attrs { replace: true } } peers { permanent_uuid: "64ed0d0908264ff0813baed49c2636e1" member_type: VOTER last_known_addr { host: "127.16.94.132" port: 42903 } attrs { promote: false } }
I20240503 14:12:29.034056 18816 raft_consensus.cc:1261] T 340236565cbb4796ad23ed5d13c9f14e P 64ed0d0908264ff0813baed49c2636e1 [term 2 LEARNER]: Refusing update from remote peer dcda0acf1e7f40369d82f940edc2becd: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 14:12:29.034758 18871 consensus_queue.cc:1035] T 340236565cbb4796ad23ed5d13c9f14e P dcda0acf1e7f40369d82f940edc2becd [LEADER]: Connected to new peer: Peer: permanent_uuid: "64ed0d0908264ff0813baed49c2636e1" member_type: VOTER last_known_addr { host: "127.16.94.132" port: 42903 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:12:29.035187 18672 raft_consensus.cc:1261] T 340236565cbb4796ad23ed5d13c9f14e P e53c5f1c8f6f4e90ad01fb89fd81a05c [term 2 FOLLOWER]: Refusing update from remote peer dcda0acf1e7f40369d82f940edc2becd: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 14:12:29.035425 18598 raft_consensus.cc:1261] T 340236565cbb4796ad23ed5d13c9f14e P f56ca8f5d65d4597a2ac8651dae8d6b7 [term 2 FOLLOWER]: Refusing update from remote peer dcda0acf1e7f40369d82f940edc2becd: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 14:12:29.036056 18871 consensus_queue.cc:1035] T 340236565cbb4796ad23ed5d13c9f14e P dcda0acf1e7f40369d82f940edc2becd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f56ca8f5d65d4597a2ac8651dae8d6b7" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:12:29.036571 18871 consensus_queue.cc:1035] T 340236565cbb4796ad23ed5d13c9f14e P dcda0acf1e7f40369d82f940edc2becd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e53c5f1c8f6f4e90ad01fb89fd81a05c" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 46285 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:12:29.039745 18868 raft_consensus.cc:2932] T 340236565cbb4796ad23ed5d13c9f14e P dcda0acf1e7f40369d82f940edc2becd [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 64ed0d0908264ff0813baed49c2636e1 (127.16.94.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dcda0acf1e7f40369d82f940edc2becd" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 46679 } } peers { permanent_uuid: "f56ca8f5d65d4597a2ac8651dae8d6b7" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39207 } } peers { permanent_uuid: "e53c5f1c8f6f4e90ad01fb89fd81a05c" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 46285 } attrs { replace: true } } peers { permanent_uuid: "64ed0d0908264ff0813baed49c2636e1" member_type: VOTER last_known_addr { host: "127.16.94.132" port: 42903 } attrs { promote: false } } }
I20240503 14:12:29.040457 18598 raft_consensus.cc:2932] T 340236565cbb4796ad23ed5d13c9f14e P f56ca8f5d65d4597a2ac8651dae8d6b7 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 64ed0d0908264ff0813baed49c2636e1 (127.16.94.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dcda0acf1e7f40369d82f940edc2becd" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 46679 } } peers { permanent_uuid: "f56ca8f5d65d4597a2ac8651dae8d6b7" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39207 } } peers { permanent_uuid: "e53c5f1c8f6f4e90ad01fb89fd81a05c" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 46285 } attrs { replace: true } } peers { permanent_uuid: "64ed0d0908264ff0813baed49c2636e1" member_type: VOTER last_known_addr { host: "127.16.94.132" port: 42903 } attrs { promote: false } } }
I20240503 14:12:29.041141 18672 raft_consensus.cc:2932] T 340236565cbb4796ad23ed5d13c9f14e P e53c5f1c8f6f4e90ad01fb89fd81a05c [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 64ed0d0908264ff0813baed49c2636e1 (127.16.94.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dcda0acf1e7f40369d82f940edc2becd" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 46679 } } peers { permanent_uuid: "f56ca8f5d65d4597a2ac8651dae8d6b7" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39207 } } peers { permanent_uuid: "e53c5f1c8f6f4e90ad01fb89fd81a05c" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 46285 } attrs { replace: true } } peers { permanent_uuid: "64ed0d0908264ff0813baed49c2636e1" member_type: VOTER last_known_addr { host: "127.16.94.132" port: 42903 } attrs { promote: false } } }
I20240503 14:12:29.043988 18816 raft_consensus.cc:2932] T 340236565cbb4796ad23ed5d13c9f14e P 64ed0d0908264ff0813baed49c2636e1 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 64ed0d0908264ff0813baed49c2636e1 (127.16.94.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dcda0acf1e7f40369d82f940edc2becd" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 46679 } } peers { permanent_uuid: "f56ca8f5d65d4597a2ac8651dae8d6b7" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39207 } } peers { permanent_uuid: "e53c5f1c8f6f4e90ad01fb89fd81a05c" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 46285 } attrs { replace: true } } peers { permanent_uuid: "64ed0d0908264ff0813baed49c2636e1" member_type: VOTER last_known_addr { host: "127.16.94.132" port: 42903 } attrs { promote: false } } }
I20240503 14:12:29.045145 18420 catalog_manager.cc:5484] T 340236565cbb4796ad23ed5d13c9f14e P dcda0acf1e7f40369d82f940edc2becd reported cstate change: config changed from index 3 to 4, 64ed0d0908264ff0813baed49c2636e1 (127.16.94.132) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "dcda0acf1e7f40369d82f940edc2becd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dcda0acf1e7f40369d82f940edc2becd" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 46679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f56ca8f5d65d4597a2ac8651dae8d6b7" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e53c5f1c8f6f4e90ad01fb89fd81a05c" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 46285 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64ed0d0908264ff0813baed49c2636e1" member_type: VOTER last_known_addr { host: "127.16.94.132" port: 42903 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:12:29.061261 18522 consensus_queue.cc:237] T 340236565cbb4796ad23ed5d13c9f14e P dcda0acf1e7f40369d82f940edc2becd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dcda0acf1e7f40369d82f940edc2becd" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 46679 } } peers { permanent_uuid: "f56ca8f5d65d4597a2ac8651dae8d6b7" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39207 } } peers { permanent_uuid: "64ed0d0908264ff0813baed49c2636e1" member_type: VOTER last_known_addr { host: "127.16.94.132" port: 42903 } attrs { promote: false } }
I20240503 14:12:29.063114 18816 raft_consensus.cc:1261] T 340236565cbb4796ad23ed5d13c9f14e P 64ed0d0908264ff0813baed49c2636e1 [term 2 FOLLOWER]: Refusing update from remote peer dcda0acf1e7f40369d82f940edc2becd: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 14:12:29.063910 18598 raft_consensus.cc:1261] T 340236565cbb4796ad23ed5d13c9f14e P f56ca8f5d65d4597a2ac8651dae8d6b7 [term 2 FOLLOWER]: Refusing update from remote peer dcda0acf1e7f40369d82f940edc2becd: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 14:12:29.064328 18872 consensus_queue.cc:1035] T 340236565cbb4796ad23ed5d13c9f14e P dcda0acf1e7f40369d82f940edc2becd [LEADER]: Connected to new peer: Peer: permanent_uuid: "64ed0d0908264ff0813baed49c2636e1" member_type: VOTER last_known_addr { host: "127.16.94.132" port: 42903 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:12:29.064579 18868 consensus_queue.cc:1035] T 340236565cbb4796ad23ed5d13c9f14e P dcda0acf1e7f40369d82f940edc2becd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f56ca8f5d65d4597a2ac8651dae8d6b7" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:12:29.066653 18869 raft_consensus.cc:2932] T 340236565cbb4796ad23ed5d13c9f14e P dcda0acf1e7f40369d82f940edc2becd [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER e53c5f1c8f6f4e90ad01fb89fd81a05c (127.16.94.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dcda0acf1e7f40369d82f940edc2becd" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 46679 } } peers { permanent_uuid: "f56ca8f5d65d4597a2ac8651dae8d6b7" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39207 } } peers { permanent_uuid: "64ed0d0908264ff0813baed49c2636e1" member_type: VOTER last_known_addr { host: "127.16.94.132" port: 42903 } attrs { promote: false } } }
I20240503 14:12:29.067238 18598 raft_consensus.cc:2932] T 340236565cbb4796ad23ed5d13c9f14e P f56ca8f5d65d4597a2ac8651dae8d6b7 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER e53c5f1c8f6f4e90ad01fb89fd81a05c (127.16.94.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dcda0acf1e7f40369d82f940edc2becd" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 46679 } } peers { permanent_uuid: "f56ca8f5d65d4597a2ac8651dae8d6b7" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39207 } } peers { permanent_uuid: "64ed0d0908264ff0813baed49c2636e1" member_type: VOTER last_known_addr { host: "127.16.94.132" port: 42903 } attrs { promote: false } } }
I20240503 14:12:29.067080 18816 raft_consensus.cc:2932] T 340236565cbb4796ad23ed5d13c9f14e P 64ed0d0908264ff0813baed49c2636e1 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER e53c5f1c8f6f4e90ad01fb89fd81a05c (127.16.94.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dcda0acf1e7f40369d82f940edc2becd" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 46679 } } peers { permanent_uuid: "f56ca8f5d65d4597a2ac8651dae8d6b7" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39207 } } peers { permanent_uuid: "64ed0d0908264ff0813baed49c2636e1" member_type: VOTER last_known_addr { host: "127.16.94.132" port: 42903 } attrs { promote: false } } }
I20240503 14:12:29.070704 18417 catalog_manager.cc:5484] T 340236565cbb4796ad23ed5d13c9f14e P f56ca8f5d65d4597a2ac8651dae8d6b7 reported cstate change: config changed from index 4 to 5, VOTER e53c5f1c8f6f4e90ad01fb89fd81a05c (127.16.94.131) evicted. New cstate: current_term: 2 leader_uuid: "dcda0acf1e7f40369d82f940edc2becd" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dcda0acf1e7f40369d82f940edc2becd" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 46679 } } peers { permanent_uuid: "f56ca8f5d65d4597a2ac8651dae8d6b7" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39207 } } peers { permanent_uuid: "64ed0d0908264ff0813baed49c2636e1" member_type: VOTER last_known_addr { host: "127.16.94.132" port: 42903 } attrs { promote: false } } }
I20240503 14:12:29.071959 18407 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 340236565cbb4796ad23ed5d13c9f14e with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:12:29.074903 18662 tablet_service.cc:1514] Processing DeleteTablet for tablet 340236565cbb4796ad23ed5d13c9f14e with delete_type TABLET_DATA_TOMBSTONED (TS e53c5f1c8f6f4e90ad01fb89fd81a05c not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:57798
I20240503 14:12:29.075875 18882 tablet_replica.cc:330] T 340236565cbb4796ad23ed5d13c9f14e P e53c5f1c8f6f4e90ad01fb89fd81a05c: stopping tablet replica
I20240503 14:12:29.076087 18882 raft_consensus.cc:2229] T 340236565cbb4796ad23ed5d13c9f14e P e53c5f1c8f6f4e90ad01fb89fd81a05c [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:29.076450 18882 raft_consensus.cc:2258] T 340236565cbb4796ad23ed5d13c9f14e P e53c5f1c8f6f4e90ad01fb89fd81a05c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:29.077267 18882 ts_tablet_manager.cc:1875] T 340236565cbb4796ad23ed5d13c9f14e P e53c5f1c8f6f4e90ad01fb89fd81a05c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:12:29.081245 18882 ts_tablet_manager.cc:1888] T 340236565cbb4796ad23ed5d13c9f14e P e53c5f1c8f6f4e90ad01fb89fd81a05c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240503 14:12:29.081343 18882 log.cc:1196] T 340236565cbb4796ad23ed5d13c9f14e P e53c5f1c8f6f4e90ad01fb89fd81a05c: Deleting WAL directory at /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714745515241994-16762-0/minicluster-data/ts-2-root/wals/340236565cbb4796ad23ed5d13c9f14e
I20240503 14:12:29.081957 18407 catalog_manager.cc:4830] TS e53c5f1c8f6f4e90ad01fb89fd81a05c (127.16.94.131:46285): tablet 340236565cbb4796ad23ed5d13c9f14e (table test-workload [id=0cc3dddc1ea14c5a908bfcd1891b6419]) successfully deleted
I20240503 14:12:30.477697 18474 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:12:30.478255 18474 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:12:30.478343 18474 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:12:32.478821 18474 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:12:32.479365 18474 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:12:32.479452 18474 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:12:34.479877 18474 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:12:34.480471 18474 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:12:34.480582 18474 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:12:36.481056 18474 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:12:36.481586 18474 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:12:36.481748 18474 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:12:38.482163 18474 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:12:38.482705 18474 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:12:38.482828 18474 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:12:40.483278 18474 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:12:40.483826 18474 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:12:40.483920 18474 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:12:41.010480 16762 tablet_server.cc:178] TabletServer@127.16.94.129:0 shutting down...
I20240503 14:12:41.028506 16762 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 14:12:41.028731 16762 tablet_replica.cc:330] T 340236565cbb4796ad23ed5d13c9f14e P dcda0acf1e7f40369d82f940edc2becd: stopping tablet replica
I20240503 14:12:41.028901 16762 raft_consensus.cc:2229] T 340236565cbb4796ad23ed5d13c9f14e P dcda0acf1e7f40369d82f940edc2becd [term 2 LEADER]: Raft consensus shutting down.
I20240503 14:12:41.029179 16762 raft_consensus.cc:2258] T 340236565cbb4796ad23ed5d13c9f14e P dcda0acf1e7f40369d82f940edc2becd [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:41.029757 16762 tablet_replica.cc:330] T e7995e0f8d084c659b0d702fda4acf66 P dcda0acf1e7f40369d82f940edc2becd: stopping tablet replica
I20240503 14:12:41.029912 16762 raft_consensus.cc:2229] T e7995e0f8d084c659b0d702fda4acf66 P dcda0acf1e7f40369d82f940edc2becd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:41.030042 16762 raft_consensus.cc:2258] T e7995e0f8d084c659b0d702fda4acf66 P dcda0acf1e7f40369d82f940edc2becd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:41.043632 16762 tablet_server.cc:194] TabletServer@127.16.94.129:0 shutdown complete.
I20240503 14:12:41.046554 16762 tablet_server.cc:178] TabletServer@127.16.94.130:0 shutting down...
I20240503 14:12:41.054061 16762 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 14:12:41.054301 16762 tablet_replica.cc:330] T 340236565cbb4796ad23ed5d13c9f14e P f56ca8f5d65d4597a2ac8651dae8d6b7: stopping tablet replica
I20240503 14:12:41.054472 16762 raft_consensus.cc:2229] T 340236565cbb4796ad23ed5d13c9f14e P f56ca8f5d65d4597a2ac8651dae8d6b7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:41.054706 16762 raft_consensus.cc:2258] T 340236565cbb4796ad23ed5d13c9f14e P f56ca8f5d65d4597a2ac8651dae8d6b7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:41.055199 16762 tablet_replica.cc:330] T e7995e0f8d084c659b0d702fda4acf66 P f56ca8f5d65d4597a2ac8651dae8d6b7: stopping tablet replica
I20240503 14:12:41.055326 16762 raft_consensus.cc:2229] T e7995e0f8d084c659b0d702fda4acf66 P f56ca8f5d65d4597a2ac8651dae8d6b7 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:41.055581 16762 raft_consensus.cc:2258] T e7995e0f8d084c659b0d702fda4acf66 P f56ca8f5d65d4597a2ac8651dae8d6b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:41.068789 16762 tablet_server.cc:194] TabletServer@127.16.94.130:0 shutdown complete.
I20240503 14:12:41.071686 16762 tablet_server.cc:178] TabletServer@127.16.94.131:0 shutting down...
I20240503 14:12:41.079380 16762 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 14:12:41.079566 16762 tablet_replica.cc:330] T e7995e0f8d084c659b0d702fda4acf66 P e53c5f1c8f6f4e90ad01fb89fd81a05c: stopping tablet replica
I20240503 14:12:41.079725 16762 raft_consensus.cc:2229] T e7995e0f8d084c659b0d702fda4acf66 P e53c5f1c8f6f4e90ad01fb89fd81a05c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:41.079869 16762 raft_consensus.cc:2258] T e7995e0f8d084c659b0d702fda4acf66 P e53c5f1c8f6f4e90ad01fb89fd81a05c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:41.093333 16762 tablet_server.cc:194] TabletServer@127.16.94.131:0 shutdown complete.
I20240503 14:12:41.096473 16762 tablet_server.cc:178] TabletServer@127.16.94.132:0 shutting down...
I20240503 14:12:41.104812 16762 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 14:12:41.105026 16762 tablet_replica.cc:330] T 340236565cbb4796ad23ed5d13c9f14e P 64ed0d0908264ff0813baed49c2636e1: stopping tablet replica
I20240503 14:12:41.105186 16762 raft_consensus.cc:2229] T 340236565cbb4796ad23ed5d13c9f14e P 64ed0d0908264ff0813baed49c2636e1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:41.105425 16762 raft_consensus.cc:2258] T 340236565cbb4796ad23ed5d13c9f14e P 64ed0d0908264ff0813baed49c2636e1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:41.118571 16762 tablet_server.cc:194] TabletServer@127.16.94.132:0 shutdown complete.
I20240503 14:12:41.122802 16762 master.cc:537] Master@127.16.94.190:44641 shutting down...
I20240503 14:12:41.132865 16762 raft_consensus.cc:2229] T 00000000000000000000000000000000 P c157bc32c1144102a7794076facb2b25 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:41.133100 16762 raft_consensus.cc:2258] T 00000000000000000000000000000000 P c157bc32c1144102a7794076facb2b25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:41.133208 16762 tablet_replica.cc:330] T 00000000000000000000000000000000 P c157bc32c1144102a7794076facb2b25: stopping tablet replica
I20240503 14:12:41.146554 16762 master.cc:552] Master@127.16.94.190:44641 shutdown complete.
[ OK ] AutoRebalancerTest.MovesScheduledIfAddTserver (26776 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfNoTservers
I20240503 14:12:41.170089 16762 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.16.94.190:42093
I20240503 14:12:41.170542 16762 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:41.173501 18888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:41.174111 18889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:41.174803 18891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:41.175485 16762 server_base.cc:1034] running on GCE node
I20240503 14:12:41.175690 16762 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:12:41.175741 16762 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:12:41.175791 16762 hybrid_clock.cc:648] HybridClock initialized: now 1714745561175786 us; error 0 us; skew 500 ppm
I20240503 14:12:41.175927 16762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:41.177120 16762 webserver.cc:460] Webserver started at http://127.16.94.190:46043/ using document root <none> and password file <none>
I20240503 14:12:41.177322 16762 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:41.177382 16762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:41.177502 16762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:41.178138 16762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714745515241994-16762-0/minicluster-data/master-0-root/instance:
uuid: "57341b7ed10a45d09c68687a42cccc5e"
format_stamp: "Formatted at 2024-05-03 14:12:41 on dist-test-slave-v27g"
I20240503 14:12:41.180485 16762 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20240503 14:12:41.182018 18896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:41.182353 16762 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 14:12:41.182442 16762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714745515241994-16762-0/minicluster-data/master-0-root
uuid: "57341b7ed10a45d09c68687a42cccc5e"
format_stamp: "Formatted at 2024-05-03 14:12:41 on dist-test-slave-v27g"
I20240503 14:12:41.182546 16762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714745515241994-16762-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714745515241994-16762-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714745515241994-16762-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:41.191344 16762 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:41.191872 16762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:41.202005 16762 rpc_server.cc:307] RPC server started. Bound to: 127.16.94.190:42093
I20240503 14:12:41.202028 18948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.94.190:42093 every 8 connection(s)
I20240503 14:12:41.203290 18949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:41.207190 18949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57341b7ed10a45d09c68687a42cccc5e: Bootstrap starting.
I20240503 14:12:41.208211 18949 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 57341b7ed10a45d09c68687a42cccc5e: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:41.209563 18949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57341b7ed10a45d09c68687a42cccc5e: No bootstrap required, opened a new log
I20240503 14:12:41.210152 18949 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57341b7ed10a45d09c68687a42cccc5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57341b7ed10a45d09c68687a42cccc5e" member_type: VOTER }
I20240503 14:12:41.210270 18949 raft_consensus.cc:374] T 00000000000000000000000000000000 P 57341b7ed10a45d09c68687a42cccc5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:41.210302 18949 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57341b7ed10a45d09c68687a42cccc5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57341b7ed10a45d09c68687a42cccc5e, State: Initialized, Role: FOLLOWER
I20240503 14:12:41.210484 18949 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57341b7ed10a45d09c68687a42cccc5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57341b7ed10a45d09c68687a42cccc5e" member_type: VOTER }
I20240503 14:12:41.210585 18949 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57341b7ed10a45d09c68687a42cccc5e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:12:41.210618 18949 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57341b7ed10a45d09c68687a42cccc5e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:12:41.210683 18949 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57341b7ed10a45d09c68687a42cccc5e [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:41.212301 18949 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57341b7ed10a45d09c68687a42cccc5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57341b7ed10a45d09c68687a42cccc5e" member_type: VOTER }
I20240503 14:12:41.212460 18949 leader_election.cc:302] T 00000000000000000000000000000000 P 57341b7ed10a45d09c68687a42cccc5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57341b7ed10a45d09c68687a42cccc5e; no voters:
I20240503 14:12:41.212816 18949 leader_election.cc:288] T 00000000000000000000000000000000 P 57341b7ed10a45d09c68687a42cccc5e [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:12:41.212967 18952 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57341b7ed10a45d09c68687a42cccc5e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:41.213431 18952 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57341b7ed10a45d09c68687a42cccc5e [term 1 LEADER]: Becoming Leader. State: Replica: 57341b7ed10a45d09c68687a42cccc5e, State: Running, Role: LEADER
I20240503 14:12:41.213579 18949 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57341b7ed10a45d09c68687a42cccc5e [sys.catalog]: configured and running, proceeding with master startup.
I20240503 14:12:41.213702 18952 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57341b7ed10a45d09c68687a42cccc5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57341b7ed10a45d09c68687a42cccc5e" member_type: VOTER }
I20240503 14:12:41.214639 18954 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57341b7ed10a45d09c68687a42cccc5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57341b7ed10a45d09c68687a42cccc5e. Latest consensus state: current_term: 1 leader_uuid: "57341b7ed10a45d09c68687a42cccc5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57341b7ed10a45d09c68687a42cccc5e" member_type: VOTER } }
I20240503 14:12:41.214843 18954 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57341b7ed10a45d09c68687a42cccc5e [sys.catalog]: This master's current role is: LEADER
I20240503 14:12:41.218320 18953 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57341b7ed10a45d09c68687a42cccc5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "57341b7ed10a45d09c68687a42cccc5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57341b7ed10a45d09c68687a42cccc5e" member_type: VOTER } }
I20240503 14:12:41.218513 18953 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57341b7ed10a45d09c68687a42cccc5e [sys.catalog]: This master's current role is: LEADER
I20240503 14:12:41.219290 18959 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 14:12:41.219693 16762 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 14:12:41.220556 18959 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 14:12:41.223151 18959 catalog_manager.cc:1347] Generated new cluster ID: c119f6bcaf1941ed9a4b79694eabc896
I20240503 14:12:41.223225 18959 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 14:12:41.240574 18959 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 14:12:41.241281 18959 catalog_manager.cc:1504] Loading token signing keys...
I20240503 14:12:41.248297 18959 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 57341b7ed10a45d09c68687a42cccc5e: Generated new TSK 0
I20240503 14:12:41.248492 18959 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 14:12:41.251833 16762 internal_mini_cluster.cc:371] 0 TS(s) registered with all masters after 5.4159e-05s
I20240503 14:12:53.333005 16762 master.cc:537] Master@127.16.94.190:42093 shutting down...
I20240503 14:12:53.347285 16762 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 57341b7ed10a45d09c68687a42cccc5e [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:12:53.347564 16762 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 57341b7ed10a45d09c68687a42cccc5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:53.347720 16762 tablet_replica.cc:330] T 00000000000000000000000000000000 P 57341b7ed10a45d09c68687a42cccc5e: stopping tablet replica
I20240503 14:12:53.362113 16762 master.cc:552] Master@127.16.94.190:42093 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfNoTservers (12201 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfNoTablets
I20240503 14:12:53.371305 16762 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.16.94.190:35877
I20240503 14:12:53.371807 16762 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:53.375033 18972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:53.375530 18971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:53.376134 18974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:53.376791 16762 server_base.cc:1034] running on GCE node
I20240503 14:12:53.379490 16762 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:12:53.379565 16762 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:12:53.379623 16762 hybrid_clock.cc:648] HybridClock initialized: now 1714745573379605 us; error 0 us; skew 500 ppm
I20240503 14:12:53.379798 16762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:53.380970 16762 webserver.cc:460] Webserver started at http://127.16.94.190:40495/ using document root <none> and password file <none>
I20240503 14:12:53.381181 16762 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:53.381265 16762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:53.381414 16762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:53.382090 16762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714745515241994-16762-0/minicluster-data/master-0-root/instance:
uuid: "2b7447a82d7b4727a41e6aeeabbca04b"
format_stamp: "Formatted at 2024-05-03 14:12:53 on dist-test-slave-v27g"
I20240503 14:12:53.384375 16762 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:53.385787 18979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:53.386139 16762 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 14:12:53.386231 16762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714745515241994-16762-0/minicluster-data/master-0-root
uuid: "2b7447a82d7b4727a41e6aeeabbca04b"
format_stamp: "Formatted at 2024-05-03 14:12:53 on dist-test-slave-v27g"
I20240503 14:12:53.386330 16762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714745515241994-16762-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714745515241994-16762-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714745515241994-16762-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:53.401293 16762 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:53.401798 16762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:53.411279 16762 rpc_server.cc:307] RPC server started. Bound to: 127.16.94.190:35877
I20240503 14:12:53.411298 19030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.94.190:35877 every 8 connection(s)
I20240503 14:12:53.412576 19031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:53.416481 19031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b7447a82d7b4727a41e6aeeabbca04b: Bootstrap starting.
I20240503 14:12:53.417536 19031 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b7447a82d7b4727a41e6aeeabbca04b: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:53.418898 19031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b7447a82d7b4727a41e6aeeabbca04b: No bootstrap required, opened a new log
I20240503 14:12:53.419482 19031 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b7447a82d7b4727a41e6aeeabbca04b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b7447a82d7b4727a41e6aeeabbca04b" member_type: VOTER }
I20240503 14:12:53.419611 19031 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b7447a82d7b4727a41e6aeeabbca04b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:53.419690 19031 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b7447a82d7b4727a41e6aeeabbca04b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b7447a82d7b4727a41e6aeeabbca04b, State: Initialized, Role: FOLLOWER
I20240503 14:12:53.419906 19031 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b7447a82d7b4727a41e6aeeabbca04b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b7447a82d7b4727a41e6aeeabbca04b" member_type: VOTER }
I20240503 14:12:53.420050 19031 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b7447a82d7b4727a41e6aeeabbca04b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:12:53.420125 19031 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b7447a82d7b4727a41e6aeeabbca04b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:12:53.420217 19031 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b7447a82d7b4727a41e6aeeabbca04b [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:53.421983 19031 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b7447a82d7b4727a41e6aeeabbca04b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b7447a82d7b4727a41e6aeeabbca04b" member_type: VOTER }
I20240503 14:12:53.422170 19031 leader_election.cc:302] T 00000000000000000000000000000000 P 2b7447a82d7b4727a41e6aeeabbca04b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b7447a82d7b4727a41e6aeeabbca04b; no voters:
I20240503 14:12:53.422549 19031 leader_election.cc:288] T 00000000000000000000000000000000 P 2b7447a82d7b4727a41e6aeeabbca04b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:12:53.422652 19034 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b7447a82d7b4727a41e6aeeabbca04b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:53.423099 19034 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b7447a82d7b4727a41e6aeeabbca04b [term 1 LEADER]: Becoming Leader. State: Replica: 2b7447a82d7b4727a41e6aeeabbca04b, State: Running, Role: LEADER
I20240503 14:12:53.423391 19031 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b7447a82d7b4727a41e6aeeabbca04b [sys.catalog]: configured and running, proceeding with master startup.
I20240503 14:12:53.423355 19034 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b7447a82d7b4727a41e6aeeabbca04b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b7447a82d7b4727a41e6aeeabbca04b" member_type: VOTER }
I20240503 14:12:53.424028 19035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b7447a82d7b4727a41e6aeeabbca04b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b7447a82d7b4727a41e6aeeabbca04b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b7447a82d7b4727a41e6aeeabbca04b" member_type: VOTER } }
I20240503 14:12:53.424082 19036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b7447a82d7b4727a41e6aeeabbca04b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b7447a82d7b4727a41e6aeeabbca04b. Latest consensus state: current_term: 1 leader_uuid: "2b7447a82d7b4727a41e6aeeabbca04b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b7447a82d7b4727a41e6aeeabbca04b" member_type: VOTER } }
I20240503 14:12:53.424283 19035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b7447a82d7b4727a41e6aeeabbca04b [sys.catalog]: This master's current role is: LEADER
I20240503 14:12:53.424301 19036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b7447a82d7b4727a41e6aeeabbca04b [sys.catalog]: This master's current role is: LEADER
I20240503 14:12:53.428081 19039 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 14:12:53.428225 16762 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 14:12:53.429232 19039 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 14:12:53.431756 19039 catalog_manager.cc:1347] Generated new cluster ID: b2eedf686f67431f81f76cc7d132020a
I20240503 14:12:53.431825 19039 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 14:12:53.440784 19039 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 14:12:53.441505 19039 catalog_manager.cc:1504] Loading token signing keys...
I20240503 14:12:53.448596 19039 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 2b7447a82d7b4727a41e6aeeabbca04b: Generated new TSK 0
I20240503 14:12:53.448815 19039 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 14:12:53.460745 16762 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:53.463874 19052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:53.464457 19053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:53.465052 19055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:53.465824 16762 server_base.cc:1034] running on GCE node
I20240503 14:12:53.466042 16762 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:12:53.466102 16762 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:12:53.466141 16762 hybrid_clock.cc:648] HybridClock initialized: now 1714745573466129 us; error 0 us; skew 500 ppm
I20240503 14:12:53.466260 16762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:53.467347 16762 webserver.cc:460] Webserver started at http://127.16.94.129:41229/ using document root <none> and password file <none>
I20240503 14:12:53.467545 16762 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:53.467608 16762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:53.467710 16762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:53.468302 16762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714745515241994-16762-0/minicluster-data/ts-0-root/instance:
uuid: "b0f1c723dfbf44d291f29f43ffe1e452"
format_stamp: "Formatted at 2024-05-03 14:12:53 on dist-test-slave-v27g"
I20240503 14:12:53.470693 16762 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20240503 14:12:53.472172 19060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:53.472469 16762 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240503 14:12:53.472592 16762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714745515241994-16762-0/minicluster-data/ts-0-root
uuid: "b0f1c723dfbf44d291f29f43ffe1e452"
format_stamp: "Formatted at 2024-05-03 14:12:53 on dist-test-slave-v27g"
I20240503 14:12:53.472697 16762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714745515241994-16762-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714745515241994-16762-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714745515241994-16762-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:53.483294 16762 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:53.483724 16762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:53.484277 16762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:53.485071 16762 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:53.485142 16762 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:53.485242 16762 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:53.485299 16762 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:53.498383 16762 rpc_server.cc:307] RPC server started. Bound to: 127.16.94.129:35933
I20240503 14:12:53.498453 19123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.94.129:35933 every 8 connection(s)
I20240503 14:12:53.499951 16762 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:53.503099 19128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:53.503509 19129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:53.507674 16762 server_base.cc:1034] running on GCE node
W20240503 14:12:53.508229 19131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:53.508374 19124 heartbeater.cc:347] Connected to a master server at 127.16.94.190:35877
I20240503 14:12:53.508533 19124 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:53.508648 16762 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:12:53.508738 16762 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:12:53.508803 16762 hybrid_clock.cc:648] HybridClock initialized: now 1714745573508779 us; error 0 us; skew 500 ppm
I20240503 14:12:53.508905 19124 heartbeater.cc:511] Master 127.16.94.190:35877 requested a full tablet report, sending...
I20240503 14:12:53.508965 16762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:53.509855 18996 ts_manager.cc:194] Registered new tserver with Master: b0f1c723dfbf44d291f29f43ffe1e452 (127.16.94.129:35933)
I20240503 14:12:53.510304 16762 webserver.cc:460] Webserver started at http://127.16.94.130:39605/ using document root <none> and password file <none>
I20240503 14:12:53.510540 16762 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:53.510624 16762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:53.510762 16762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:53.510986 18996 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39950
I20240503 14:12:53.511445 16762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714745515241994-16762-0/minicluster-data/ts-1-root/instance:
uuid: "d9dca404feeb43f78e633aa3b25bc255"
format_stamp: "Formatted at 2024-05-03 14:12:53 on dist-test-slave-v27g"
I20240503 14:12:53.513936 16762 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20240503 14:12:53.515276 19136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:53.515604 16762 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240503 14:12:53.515714 16762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714745515241994-16762-0/minicluster-data/ts-1-root
uuid: "d9dca404feeb43f78e633aa3b25bc255"
format_stamp: "Formatted at 2024-05-03 14:12:53 on dist-test-slave-v27g"
I20240503 14:12:53.515834 16762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714745515241994-16762-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714745515241994-16762-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714745515241994-16762-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:53.561122 16762 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:53.561630 16762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:53.562219 16762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:53.562997 16762 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:53.563059 16762 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:53.563151 16762 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:53.563213 16762 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:53.574985 16762 rpc_server.cc:307] RPC server started. Bound to: 127.16.94.130:39569
I20240503 14:12:53.575017 19198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.94.130:39569 every 8 connection(s)
I20240503 14:12:53.576402 16762 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:53.579795 19203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:53.580268 19202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:53.582937 19199 heartbeater.cc:347] Connected to a master server at 127.16.94.190:35877
I20240503 14:12:53.583070 19199 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:53.583170 16762 server_base.cc:1034] running on GCE node
I20240503 14:12:53.583437 19199 heartbeater.cc:511] Master 127.16.94.190:35877 requested a full tablet report, sending...
W20240503 14:12:53.583717 19206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:53.584162 16762 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:12:53.584249 16762 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:12:53.584317 18996 ts_manager.cc:194] Registered new tserver with Master: d9dca404feeb43f78e633aa3b25bc255 (127.16.94.130:39569)
I20240503 14:12:53.584317 16762 hybrid_clock.cc:648] HybridClock initialized: now 1714745573584289 us; error 0 us; skew 500 ppm
I20240503 14:12:53.584520 16762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:53.585062 18996 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39952
I20240503 14:12:53.585855 16762 webserver.cc:460] Webserver started at http://127.16.94.131:33373/ using document root <none> and password file <none>
I20240503 14:12:53.586107 16762 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:53.586202 16762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:53.586335 16762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:53.586910 16762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714745515241994-16762-0/minicluster-data/ts-2-root/instance:
uuid: "2210e86d11154f37ab65b702f244b070"
format_stamp: "Formatted at 2024-05-03 14:12:53 on dist-test-slave-v27g"
I20240503 14:12:53.589075 16762 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20240503 14:12:53.590371 19211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:53.590706 16762 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 14:12:53.590816 16762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714745515241994-16762-0/minicluster-data/ts-2-root
uuid: "2210e86d11154f37ab65b702f244b070"
format_stamp: "Formatted at 2024-05-03 14:12:53 on dist-test-slave-v27g"
I20240503 14:12:53.590924 16762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714745515241994-16762-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714745515241994-16762-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714745515241994-16762-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:53.597043 16762 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:53.597451 16762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:53.597937 16762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:53.598647 16762 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:53.598723 16762 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:53.598817 16762 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:53.598871 16762 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:53.610910 16762 rpc_server.cc:307] RPC server started. Bound to: 127.16.94.131:46745
I20240503 14:12:53.610934 19273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.94.131:46745 every 8 connection(s)
I20240503 14:12:53.616984 19274 heartbeater.cc:347] Connected to a master server at 127.16.94.190:35877
I20240503 14:12:53.617125 19274 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:53.617447 19274 heartbeater.cc:511] Master 127.16.94.190:35877 requested a full tablet report, sending...
I20240503 14:12:53.618247 18996 ts_manager.cc:194] Registered new tserver with Master: 2210e86d11154f37ab65b702f244b070 (127.16.94.131:46745)
I20240503 14:12:53.618482 16762 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.006715098s
I20240503 14:12:53.618993 18996 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39960
I20240503 14:12:54.512209 19124 heartbeater.cc:503] Master 127.16.94.190:35877 was elected leader, sending a full tablet report...
I20240503 14:12:54.586545 19199 heartbeater.cc:503] Master 127.16.94.190:35877 was elected leader, sending a full tablet report...
I20240503 14:12:54.620189 19274 heartbeater.cc:503] Master 127.16.94.190:35877 was elected leader, sending a full tablet report...
I20240503 14:12:55.427698 19049 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:12:57.428082 19049 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:12:59.428563 19049 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:13:01.428978 19049 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:13:03.429376 19049 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:13:05.429832 19049 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:13:05.700489 16762 tablet_server.cc:178] TabletServer@127.16.94.129:0 shutting down...
I20240503 14:13:05.717731 16762 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 14:13:05.731564 16762 tablet_server.cc:194] TabletServer@127.16.94.129:0 shutdown complete.
I20240503 14:13:05.734527 16762 tablet_server.cc:178] TabletServer@127.16.94.130:0 shutting down...
I20240503 14:13:05.742715 16762 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 14:13:05.755343 16762 tablet_server.cc:194] TabletServer@127.16.94.130:0 shutdown complete.
I20240503 14:13:05.757872 16762 tablet_server.cc:178] TabletServer@127.16.94.131:0 shutting down...
I20240503 14:13:05.766913 16762 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 14:13:05.779309 16762 tablet_server.cc:194] TabletServer@127.16.94.131:0 shutdown complete.
I20240503 14:13:05.781715 16762 master.cc:537] Master@127.16.94.190:35877 shutting down...
I20240503 14:13:05.790295 16762 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 2b7447a82d7b4727a41e6aeeabbca04b [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:05.790508 16762 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 2b7447a82d7b4727a41e6aeeabbca04b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:05.790625 16762 tablet_replica.cc:330] T 00000000000000000000000000000000 P 2b7447a82d7b4727a41e6aeeabbca04b: stopping tablet replica
I20240503 14:13:05.803531 16762 master.cc:552] Master@127.16.94.190:35877 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfNoTablets (12441 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne
I20240503 14:13:05.812399 16762 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.16.94.190:43231
I20240503 14:13:05.812898 16762 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:13:05.815670 19278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:05.816246 19279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:05.816777 19281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:13:05.817301 16762 server_base.cc:1034] running on GCE node
I20240503 14:13:05.817530 16762 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:13:05.817602 16762 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:13:05.817688 16762 hybrid_clock.cc:648] HybridClock initialized: now 1714745585817669 us; error 0 us; skew 500 ppm
I20240503 14:13:05.817857 16762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:13:05.819003 16762 webserver.cc:460] Webserver started at http://127.16.94.190:34879/ using document root <none> and password file <none>
I20240503 14:13:05.819206 16762 fs_manager.cc:359] Metadata directory not provided
I20240503 14:13:05.819283 16762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:13:05.819417 16762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:13:05.820056 16762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714745515241994-16762-0/minicluster-data/master-0-root/instance:
uuid: "57be60b30b274fce8f0c2016ed5ff7ea"
format_stamp: "Formatted at 2024-05-03 14:13:05 on dist-test-slave-v27g"
I20240503 14:13:05.822628 16762 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 14:13:05.824112 19286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:05.824472 16762 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240503 14:13:05.824565 16762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714745515241994-16762-0/minicluster-data/master-0-root
uuid: "57be60b30b274fce8f0c2016ed5ff7ea"
format_stamp: "Formatted at 2024-05-03 14:13:05 on dist-test-slave-v27g"
I20240503 14:13:05.824659 16762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714745515241994-16762-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714745515241994-16762-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714745515241994-16762-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:13:05.843658 16762 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:13:05.844215 16762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:13:05.854997 16762 rpc_server.cc:307] RPC server started. Bound to: 127.16.94.190:43231
I20240503 14:13:05.855015 19337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.94.190:43231 every 8 connection(s)
I20240503 14:13:05.856197 19338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:05.860224 19338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57be60b30b274fce8f0c2016ed5ff7ea: Bootstrap starting.
I20240503 14:13:05.861356 19338 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 57be60b30b274fce8f0c2016ed5ff7ea: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:05.862808 19338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57be60b30b274fce8f0c2016ed5ff7ea: No bootstrap required, opened a new log
I20240503 14:13:05.863322 19338 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57be60b30b274fce8f0c2016ed5ff7ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57be60b30b274fce8f0c2016ed5ff7ea" member_type: VOTER }
I20240503 14:13:05.863451 19338 raft_consensus.cc:374] T 00000000000000000000000000000000 P 57be60b30b274fce8f0c2016ed5ff7ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:05.863509 19338 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57be60b30b274fce8f0c2016ed5ff7ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57be60b30b274fce8f0c2016ed5ff7ea, State: Initialized, Role: FOLLOWER
I20240503 14:13:05.863703 19338 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57be60b30b274fce8f0c2016ed5ff7ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57be60b30b274fce8f0c2016ed5ff7ea" member_type: VOTER }
I20240503 14:13:05.863830 19338 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57be60b30b274fce8f0c2016ed5ff7ea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:13:05.863883 19338 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57be60b30b274fce8f0c2016ed5ff7ea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:13:05.863950 19338 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57be60b30b274fce8f0c2016ed5ff7ea [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:05.865564 19338 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57be60b30b274fce8f0c2016ed5ff7ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57be60b30b274fce8f0c2016ed5ff7ea" member_type: VOTER }
I20240503 14:13:05.865772 19338 leader_election.cc:302] T 00000000000000000000000000000000 P 57be60b30b274fce8f0c2016ed5ff7ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57be60b30b274fce8f0c2016ed5ff7ea; no voters:
I20240503 14:13:05.866104 19338 leader_election.cc:288] T 00000000000000000000000000000000 P 57be60b30b274fce8f0c2016ed5ff7ea [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:13:05.866233 19341 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57be60b30b274fce8f0c2016ed5ff7ea [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:05.866690 19341 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57be60b30b274fce8f0c2016ed5ff7ea [term 1 LEADER]: Becoming Leader. State: Replica: 57be60b30b274fce8f0c2016ed5ff7ea, State: Running, Role: LEADER
I20240503 14:13:05.866943 19338 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57be60b30b274fce8f0c2016ed5ff7ea [sys.catalog]: configured and running, proceeding with master startup.
I20240503 14:13:05.866954 19341 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57be60b30b274fce8f0c2016ed5ff7ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57be60b30b274fce8f0c2016ed5ff7ea" member_type: VOTER }
I20240503 14:13:05.867918 19342 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57be60b30b274fce8f0c2016ed5ff7ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "57be60b30b274fce8f0c2016ed5ff7ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57be60b30b274fce8f0c2016ed5ff7ea" member_type: VOTER } }
I20240503 14:13:05.867992 19343 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57be60b30b274fce8f0c2016ed5ff7ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57be60b30b274fce8f0c2016ed5ff7ea. Latest consensus state: current_term: 1 leader_uuid: "57be60b30b274fce8f0c2016ed5ff7ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57be60b30b274fce8f0c2016ed5ff7ea" member_type: VOTER } }
I20240503 14:13:05.868217 19343 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57be60b30b274fce8f0c2016ed5ff7ea [sys.catalog]: This master's current role is: LEADER
I20240503 14:13:05.868686 19342 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57be60b30b274fce8f0c2016ed5ff7ea [sys.catalog]: This master's current role is: LEADER
I20240503 14:13:05.869189 19347 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 14:13:05.870643 19347 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 14:13:05.872330 16762 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 14:13:05.873315 19347 catalog_manager.cc:1347] Generated new cluster ID: 75c588cb72e142f0858bbebb05de05a8
I20240503 14:13:05.873404 19347 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 14:13:05.894455 19347 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 14:13:05.895190 19347 catalog_manager.cc:1504] Loading token signing keys...
I20240503 14:13:05.906864 19347 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 57be60b30b274fce8f0c2016ed5ff7ea: Generated new TSK 0
I20240503 14:13:05.907086 19347 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 14:13:05.937434 16762 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:13:05.940651 19359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:05.941262 19360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:05.941771 19362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:13:05.942318 16762 server_base.cc:1034] running on GCE node
I20240503 14:13:05.942584 16762 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:13:05.942643 16762 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:13:05.942715 16762 hybrid_clock.cc:648] HybridClock initialized: now 1714745585942697 us; error 0 us; skew 500 ppm
I20240503 14:13:05.942865 16762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:13:05.944082 16762 webserver.cc:460] Webserver started at http://127.16.94.129:43197/ using document root <none> and password file <none>
I20240503 14:13:05.944294 16762 fs_manager.cc:359] Metadata directory not provided
I20240503 14:13:05.944372 16762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:13:05.944504 16762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:13:05.945154 16762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714745515241994-16762-0/minicluster-data/ts-0-root/instance:
uuid: "0e1a6a35fa24463391f1208fbd73e35c"
format_stamp: "Formatted at 2024-05-03 14:13:05 on dist-test-slave-v27g"
I20240503 14:13:05.947587 16762 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 14:13:05.949111 19367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:05.949429 16762 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 14:13:05.949537 16762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714745515241994-16762-0/minicluster-data/ts-0-root
uuid: "0e1a6a35fa24463391f1208fbd73e35c"
format_stamp: "Formatted at 2024-05-03 14:13:05 on dist-test-slave-v27g"
I20240503 14:13:05.949677 16762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714745515241994-16762-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714745515241994-16762-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714745515241994-16762-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:13:05.973263 16762 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:13:05.973894 16762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:13:05.974436 16762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:13:05.975327 16762 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:13:05.975397 16762 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:05.975525 16762 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:13:05.975584 16762 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:05.988045 16762 rpc_server.cc:307] RPC server started. Bound to: 127.16.94.129:45249
I20240503 14:13:05.988065 19429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.94.129:45249 every 8 connection(s)
I20240503 14:13:05.989806 16762 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:13:05.993446 19434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:05.994026 19435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:05.994504 19437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:13:05.995421 16762 server_base.cc:1034] running on GCE node
I20240503 14:13:05.995764 16762 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:13:05.995879 16762 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:13:05.995946 16762 hybrid_clock.cc:648] HybridClock initialized: now 1714745585995924 us; error 0 us; skew 500 ppm
I20240503 14:13:05.996126 16762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:13:05.997083 19430 heartbeater.cc:347] Connected to a master server at 127.16.94.190:43231
I20240503 14:13:05.997210 19430 heartbeater.cc:464] Registering TS with master...
I20240503 14:13:05.997534 19430 heartbeater.cc:511] Master 127.16.94.190:43231 requested a full tablet report, sending...
I20240503 14:13:05.997942 16762 webserver.cc:460] Webserver started at http://127.16.94.130:42473/ using document root <none> and password file <none>
I20240503 14:13:05.998211 16762 fs_manager.cc:359] Metadata directory not provided
I20240503 14:13:05.998302 16762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:13:05.998440 16762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:13:05.998481 19303 ts_manager.cc:194] Registered new tserver with Master: 0e1a6a35fa24463391f1208fbd73e35c (127.16.94.129:45249)
I20240503 14:13:05.999150 16762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714745515241994-16762-0/minicluster-data/ts-1-root/instance:
uuid: "e92d07359fbb434bbe6b3db778ad74be"
format_stamp: "Formatted at 2024-05-03 14:13:05 on dist-test-slave-v27g"
I20240503 14:13:05.999677 19303 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52942
I20240503 14:13:06.001643 16762 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20240503 14:13:06.003163 19442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:06.003471 16762 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 14:13:06.003585 16762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714745515241994-16762-0/minicluster-data/ts-1-root
uuid: "e92d07359fbb434bbe6b3db778ad74be"
format_stamp: "Formatted at 2024-05-03 14:13:05 on dist-test-slave-v27g"
I20240503 14:13:06.003738 16762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714745515241994-16762-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714745515241994-16762-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714745515241994-16762-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:13:06.016635 16762 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:13:06.017167 16762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:13:06.017645 16762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:13:06.018467 16762 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:13:06.018535 16762 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:06.018632 16762 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:13:06.018687 16762 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:06.030926 16762 rpc_server.cc:307] RPC server started. Bound to: 127.16.94.130:45431
I20240503 14:13:06.030959 19504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.94.130:45431 every 8 connection(s)
I20240503 14:13:06.032579 16762 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:13:06.036365 19508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:06.036803 19509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:13:06.037858 16762 server_base.cc:1034] running on GCE node
W20240503 14:13:06.038347 19511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:13:06.038743 16762 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:13:06.038826 16762 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:13:06.038921 16762 hybrid_clock.cc:648] HybridClock initialized: now 1714745586038898 us; error 0 us; skew 500 ppm
I20240503 14:13:06.039072 19505 heartbeater.cc:347] Connected to a master server at 127.16.94.190:43231
I20240503 14:13:06.039103 16762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:13:06.039244 19505 heartbeater.cc:464] Registering TS with master...
I20240503 14:13:06.039645 19505 heartbeater.cc:511] Master 127.16.94.190:43231 requested a full tablet report, sending...
I20240503 14:13:06.040452 16762 webserver.cc:460] Webserver started at http://127.16.94.131:35577/ using document root <none> and password file <none>
I20240503 14:13:06.040625 19303 ts_manager.cc:194] Registered new tserver with Master: e92d07359fbb434bbe6b3db778ad74be (127.16.94.130:45431)
I20240503 14:13:06.040706 16762 fs_manager.cc:359] Metadata directory not provided
I20240503 14:13:06.040812 16762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:13:06.040956 16762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:13:06.041431 19303 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52946
I20240503 14:13:06.041591 16762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714745515241994-16762-0/minicluster-data/ts-2-root/instance:
uuid: "d34bda967218462d8e840b14cd36cd95"
format_stamp: "Formatted at 2024-05-03 14:13:06 on dist-test-slave-v27g"
I20240503 14:13:06.044286 16762 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:06.045743 19516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:06.046034 16762 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 14:13:06.046142 16762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714745515241994-16762-0/minicluster-data/ts-2-root
uuid: "d34bda967218462d8e840b14cd36cd95"
format_stamp: "Formatted at 2024-05-03 14:13:06 on dist-test-slave-v27g"
I20240503 14:13:06.046252 16762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714745515241994-16762-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714745515241994-16762-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714745515241994-16762-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:13:06.058241 16762 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:13:06.058768 16762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:13:06.059257 16762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:13:06.059990 16762 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:13:06.060060 16762 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:06.060153 16762 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:13:06.060207 16762 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:06.075596 16762 rpc_server.cc:307] RPC server started. Bound to: 127.16.94.131:39493
I20240503 14:13:06.075623 19578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.94.131:39493 every 8 connection(s)
I20240503 14:13:06.077162 16762 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:13:06.080663 19582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:06.081210 19583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:13:06.081881 16762 server_base.cc:1034] running on GCE node
W20240503 14:13:06.082271 19585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:13:06.082641 16762 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:13:06.082743 16762 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:13:06.082811 16762 hybrid_clock.cc:648] HybridClock initialized: now 1714745586082783 us; error 0 us; skew 500 ppm
I20240503 14:13:06.083010 16762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:13:06.084553 19579 heartbeater.cc:347] Connected to a master server at 127.16.94.190:43231
I20240503 14:13:06.084707 19579 heartbeater.cc:464] Registering TS with master...
I20240503 14:13:06.084705 16762 webserver.cc:460] Webserver started at http://127.16.94.132:38951/ using document root <none> and password file <none>
I20240503 14:13:06.084976 16762 fs_manager.cc:359] Metadata directory not provided
I20240503 14:13:06.085075 16762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:13:06.085160 19579 heartbeater.cc:511] Master 127.16.94.190:43231 requested a full tablet report, sending...
I20240503 14:13:06.085218 16762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:13:06.086032 16762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714745515241994-16762-0/minicluster-data/ts-3-root/instance:
uuid: "a2e6cd2a63624009ab7d31ee288ea4af"
format_stamp: "Formatted at 2024-05-03 14:13:06 on dist-test-slave-v27g"
I20240503 14:13:06.086148 19303 ts_manager.cc:194] Registered new tserver with Master: d34bda967218462d8e840b14cd36cd95 (127.16.94.131:39493)
I20240503 14:13:06.086911 19303 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52954
I20240503 14:13:06.088840 16762 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 14:13:06.090364 19590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:06.090704 16762 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 14:13:06.090816 16762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714745515241994-16762-0/minicluster-data/ts-3-root
uuid: "a2e6cd2a63624009ab7d31ee288ea4af"
format_stamp: "Formatted at 2024-05-03 14:13:06 on dist-test-slave-v27g"
I20240503 14:13:06.090925 16762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714745515241994-16762-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714745515241994-16762-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714745515241994-16762-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:13:06.101125 16762 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:13:06.101728 16762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:13:06.102260 16762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:13:06.103097 16762 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:13:06.103168 16762 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:06.103269 16762 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:13:06.103334 16762 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:06.116497 16762 rpc_server.cc:307] RPC server started. Bound to: 127.16.94.132:39913
I20240503 14:13:06.116519 19652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.94.132:39913 every 8 connection(s)
I20240503 14:13:06.123299 19653 heartbeater.cc:347] Connected to a master server at 127.16.94.190:43231
I20240503 14:13:06.123435 19653 heartbeater.cc:464] Registering TS with master...
I20240503 14:13:06.123796 19653 heartbeater.cc:511] Master 127.16.94.190:43231 requested a full tablet report, sending...
I20240503 14:13:06.124572 19303 ts_manager.cc:194] Registered new tserver with Master: a2e6cd2a63624009ab7d31ee288ea4af (127.16.94.132:39913)
I20240503 14:13:06.125304 16762 internal_mini_cluster.cc:371] 4 TS(s) registered with all masters after 0.007953308s
I20240503 14:13:06.125314 19303 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52966
I20240503 14:13:07.000885 19430 heartbeater.cc:503] Master 127.16.94.190:43231 was elected leader, sending a full tablet report...
I20240503 14:13:07.042871 19505 heartbeater.cc:503] Master 127.16.94.190:43231 was elected leader, sending a full tablet report...
I20240503 14:13:07.088287 19579 heartbeater.cc:503] Master 127.16.94.190:43231 was elected leader, sending a full tablet report...
I20240503 14:13:07.126657 19653 heartbeater.cc:503] Master 127.16.94.190:43231 was elected leader, sending a full tablet report...
I20240503 14:13:07.180044 16762 test_util.cc:268] Using random seed: 1369088504
I20240503 14:13:07.190985 19303 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52982:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:13:07.218600 19544 tablet_service.cc:1467] Processing CreateTablet for tablet 1b9da24397c1439fbf4d7ecd26ff89f5 (DEFAULT_TABLE table=test-workload [id=bd59783e7cf947e28ab06c575344447d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:07.219111 19544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b9da24397c1439fbf4d7ecd26ff89f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:07.219077 19395 tablet_service.cc:1467] Processing CreateTablet for tablet 1b9da24397c1439fbf4d7ecd26ff89f5 (DEFAULT_TABLE table=test-workload [id=bd59783e7cf947e28ab06c575344447d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:07.219408 19395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b9da24397c1439fbf4d7ecd26ff89f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:07.222234 19470 tablet_service.cc:1467] Processing CreateTablet for tablet 1b9da24397c1439fbf4d7ecd26ff89f5 (DEFAULT_TABLE table=test-workload [id=bd59783e7cf947e28ab06c575344447d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:07.222549 19470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b9da24397c1439fbf4d7ecd26ff89f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:07.223875 19674 tablet_bootstrap.cc:492] T 1b9da24397c1439fbf4d7ecd26ff89f5 P 0e1a6a35fa24463391f1208fbd73e35c: Bootstrap starting.
I20240503 14:13:07.224481 19675 tablet_bootstrap.cc:492] T 1b9da24397c1439fbf4d7ecd26ff89f5 P d34bda967218462d8e840b14cd36cd95: Bootstrap starting.
I20240503 14:13:07.225574 19674 tablet_bootstrap.cc:654] T 1b9da24397c1439fbf4d7ecd26ff89f5 P 0e1a6a35fa24463391f1208fbd73e35c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:07.226115 19675 tablet_bootstrap.cc:654] T 1b9da24397c1439fbf4d7ecd26ff89f5 P d34bda967218462d8e840b14cd36cd95: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:07.227468 19674 tablet_bootstrap.cc:492] T 1b9da24397c1439fbf4d7ecd26ff89f5 P 0e1a6a35fa24463391f1208fbd73e35c: No bootstrap required, opened a new log
I20240503 14:13:07.227630 19674 ts_tablet_manager.cc:1367] T 1b9da24397c1439fbf4d7ecd26ff89f5 P 0e1a6a35fa24463391f1208fbd73e35c: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:13:07.228195 19678 tablet_bootstrap.cc:492] T 1b9da24397c1439fbf4d7ecd26ff89f5 P e92d07359fbb434bbe6b3db778ad74be: Bootstrap starting.
I20240503 14:13:07.228536 19674 raft_consensus.cc:348] T 1b9da24397c1439fbf4d7ecd26ff89f5 P 0e1a6a35fa24463391f1208fbd73e35c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e92d07359fbb434bbe6b3db778ad74be" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 45431 } } peers { permanent_uuid: "d34bda967218462d8e840b14cd36cd95" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39493 } } peers { permanent_uuid: "0e1a6a35fa24463391f1208fbd73e35c" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 45249 } }
I20240503 14:13:07.228724 19674 raft_consensus.cc:374] T 1b9da24397c1439fbf4d7ecd26ff89f5 P 0e1a6a35fa24463391f1208fbd73e35c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:07.228806 19674 raft_consensus.cc:729] T 1b9da24397c1439fbf4d7ecd26ff89f5 P 0e1a6a35fa24463391f1208fbd73e35c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e1a6a35fa24463391f1208fbd73e35c, State: Initialized, Role: FOLLOWER
I20240503 14:13:07.229070 19674 consensus_queue.cc:260] T 1b9da24397c1439fbf4d7ecd26ff89f5 P 0e1a6a35fa24463391f1208fbd73e35c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e92d07359fbb434bbe6b3db778ad74be" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 45431 } } peers { permanent_uuid: "d34bda967218462d8e840b14cd36cd95" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39493 } } peers { permanent_uuid: "0e1a6a35fa24463391f1208fbd73e35c" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 45249 } }
I20240503 14:13:07.229377 19675 tablet_bootstrap.cc:492] T 1b9da24397c1439fbf4d7ecd26ff89f5 P d34bda967218462d8e840b14cd36cd95: No bootstrap required, opened a new log
I20240503 14:13:07.229557 19675 ts_tablet_manager.cc:1367] T 1b9da24397c1439fbf4d7ecd26ff89f5 P d34bda967218462d8e840b14cd36cd95: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20240503 14:13:07.229817 19674 ts_tablet_manager.cc:1398] T 1b9da24397c1439fbf4d7ecd26ff89f5 P 0e1a6a35fa24463391f1208fbd73e35c: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 14:13:07.230258 19675 raft_consensus.cc:348] T 1b9da24397c1439fbf4d7ecd26ff89f5 P d34bda967218462d8e840b14cd36cd95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e92d07359fbb434bbe6b3db778ad74be" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 45431 } } peers { permanent_uuid: "d34bda967218462d8e840b14cd36cd95" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39493 } } peers { permanent_uuid: "0e1a6a35fa24463391f1208fbd73e35c" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 45249 } }
I20240503 14:13:07.230463 19675 raft_consensus.cc:374] T 1b9da24397c1439fbf4d7ecd26ff89f5 P d34bda967218462d8e840b14cd36cd95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:07.230544 19675 raft_consensus.cc:729] T 1b9da24397c1439fbf4d7ecd26ff89f5 P d34bda967218462d8e840b14cd36cd95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d34bda967218462d8e840b14cd36cd95, State: Initialized, Role: FOLLOWER
I20240503 14:13:07.230552 19678 tablet_bootstrap.cc:654] T 1b9da24397c1439fbf4d7ecd26ff89f5 P e92d07359fbb434bbe6b3db778ad74be: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:07.230756 19675 consensus_queue.cc:260] T 1b9da24397c1439fbf4d7ecd26ff89f5 P d34bda967218462d8e840b14cd36cd95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e92d07359fbb434bbe6b3db778ad74be" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 45431 } } peers { permanent_uuid: "d34bda967218462d8e840b14cd36cd95" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39493 } } peers { permanent_uuid: "0e1a6a35fa24463391f1208fbd73e35c" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 45249 } }
I20240503 14:13:07.231417 19675 ts_tablet_manager.cc:1398] T 1b9da24397c1439fbf4d7ecd26ff89f5 P d34bda967218462d8e840b14cd36cd95: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:07.232523 19678 tablet_bootstrap.cc:492] T 1b9da24397c1439fbf4d7ecd26ff89f5 P e92d07359fbb434bbe6b3db778ad74be: No bootstrap required, opened a new log
I20240503 14:13:07.232707 19678 ts_tablet_manager.cc:1367] T 1b9da24397c1439fbf4d7ecd26ff89f5 P e92d07359fbb434bbe6b3db778ad74be: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 14:13:07.233302 19678 raft_consensus.cc:348] T 1b9da24397c1439fbf4d7ecd26ff89f5 P e92d07359fbb434bbe6b3db778ad74be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e92d07359fbb434bbe6b3db778ad74be" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 45431 } } peers { permanent_uuid: "d34bda967218462d8e840b14cd36cd95" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39493 } } peers { permanent_uuid: "0e1a6a35fa24463391f1208fbd73e35c" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 45249 } }
I20240503 14:13:07.233546 19678 raft_consensus.cc:374] T 1b9da24397c1439fbf4d7ecd26ff89f5 P e92d07359fbb434bbe6b3db778ad74be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:07.233680 19678 raft_consensus.cc:729] T 1b9da24397c1439fbf4d7ecd26ff89f5 P e92d07359fbb434bbe6b3db778ad74be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e92d07359fbb434bbe6b3db778ad74be, State: Initialized, Role: FOLLOWER
I20240503 14:13:07.233906 19678 consensus_queue.cc:260] T 1b9da24397c1439fbf4d7ecd26ff89f5 P e92d07359fbb434bbe6b3db778ad74be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e92d07359fbb434bbe6b3db778ad74be" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 45431 } } peers { permanent_uuid: "d34bda967218462d8e840b14cd36cd95" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39493 } } peers { permanent_uuid: "0e1a6a35fa24463391f1208fbd73e35c" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 45249 } }
I20240503 14:13:07.234613 19678 ts_tablet_manager.cc:1398] T 1b9da24397c1439fbf4d7ecd26ff89f5 P e92d07359fbb434bbe6b3db778ad74be: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20240503 14:13:07.240128 19431 tablet.cc:2369] T 1b9da24397c1439fbf4d7ecd26ff89f5 P 0e1a6a35fa24463391f1208fbd73e35c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:13:07.392764 19680 raft_consensus.cc:482] T 1b9da24397c1439fbf4d7ecd26ff89f5 P d34bda967218462d8e840b14cd36cd95 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:07.393028 19680 raft_consensus.cc:504] T 1b9da24397c1439fbf4d7ecd26ff89f5 P d34bda967218462d8e840b14cd36cd95 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e92d07359fbb434bbe6b3db778ad74be" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 45431 } } peers { permanent_uuid: "d34bda967218462d8e840b14cd36cd95" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39493 } } peers { permanent_uuid: "0e1a6a35fa24463391f1208fbd73e35c" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 45249 } }
I20240503 14:13:07.394711 19680 leader_election.cc:288] T 1b9da24397c1439fbf4d7ecd26ff89f5 P d34bda967218462d8e840b14cd36cd95 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e92d07359fbb434bbe6b3db778ad74be (127.16.94.130:45431), 0e1a6a35fa24463391f1208fbd73e35c (127.16.94.129:45249)
I20240503 14:13:07.399354 19405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b9da24397c1439fbf4d7ecd26ff89f5" candidate_uuid: "d34bda967218462d8e840b14cd36cd95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e1a6a35fa24463391f1208fbd73e35c" is_pre_election: true
I20240503 14:13:07.399355 19480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b9da24397c1439fbf4d7ecd26ff89f5" candidate_uuid: "d34bda967218462d8e840b14cd36cd95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e92d07359fbb434bbe6b3db778ad74be" is_pre_election: true
I20240503 14:13:07.399540 19405 raft_consensus.cc:2454] T 1b9da24397c1439fbf4d7ecd26ff89f5 P 0e1a6a35fa24463391f1208fbd73e35c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d34bda967218462d8e840b14cd36cd95 in term 0.
I20240503 14:13:07.399574 19480 raft_consensus.cc:2454] T 1b9da24397c1439fbf4d7ecd26ff89f5 P e92d07359fbb434bbe6b3db778ad74be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d34bda967218462d8e840b14cd36cd95 in term 0.
I20240503 14:13:07.399916 19517 leader_election.cc:302] T 1b9da24397c1439fbf4d7ecd26ff89f5 P d34bda967218462d8e840b14cd36cd95 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e1a6a35fa24463391f1208fbd73e35c, d34bda967218462d8e840b14cd36cd95; no voters:
I20240503 14:13:07.400121 19680 raft_consensus.cc:2781] T 1b9da24397c1439fbf4d7ecd26ff89f5 P d34bda967218462d8e840b14cd36cd95 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:07.400184 19680 raft_consensus.cc:482] T 1b9da24397c1439fbf4d7ecd26ff89f5 P d34bda967218462d8e840b14cd36cd95 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:07.400228 19680 raft_consensus.cc:3037] T 1b9da24397c1439fbf4d7ecd26ff89f5 P d34bda967218462d8e840b14cd36cd95 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:07.401944 19680 raft_consensus.cc:504] T 1b9da24397c1439fbf4d7ecd26ff89f5 P d34bda967218462d8e840b14cd36cd95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e92d07359fbb434bbe6b3db778ad74be" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 45431 } } peers { permanent_uuid: "d34bda967218462d8e840b14cd36cd95" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39493 } } peers { permanent_uuid: "0e1a6a35fa24463391f1208fbd73e35c" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 45249 } }
I20240503 14:13:07.402452 19680 leader_election.cc:288] T 1b9da24397c1439fbf4d7ecd26ff89f5 P d34bda967218462d8e840b14cd36cd95 [CANDIDATE]: Term 1 election: Requested vote from peers e92d07359fbb434bbe6b3db778ad74be (127.16.94.130:45431), 0e1a6a35fa24463391f1208fbd73e35c (127.16.94.129:45249)
I20240503 14:13:07.402861 19480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b9da24397c1439fbf4d7ecd26ff89f5" candidate_uuid: "d34bda967218462d8e840b14cd36cd95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e92d07359fbb434bbe6b3db778ad74be"
I20240503 14:13:07.402861 19405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b9da24397c1439fbf4d7ecd26ff89f5" candidate_uuid: "d34bda967218462d8e840b14cd36cd95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e1a6a35fa24463391f1208fbd73e35c"
I20240503 14:13:07.403067 19405 raft_consensus.cc:3037] T 1b9da24397c1439fbf4d7ecd26ff89f5 P 0e1a6a35fa24463391f1208fbd73e35c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:07.403067 19480 raft_consensus.cc:3037] T 1b9da24397c1439fbf4d7ecd26ff89f5 P e92d07359fbb434bbe6b3db778ad74be [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:07.405735 19480 raft_consensus.cc:2454] T 1b9da24397c1439fbf4d7ecd26ff89f5 P e92d07359fbb434bbe6b3db778ad74be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d34bda967218462d8e840b14cd36cd95 in term 1.
I20240503 14:13:07.405735 19405 raft_consensus.cc:2454] T 1b9da24397c1439fbf4d7ecd26ff89f5 P 0e1a6a35fa24463391f1208fbd73e35c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d34bda967218462d8e840b14cd36cd95 in term 1.
I20240503 14:13:07.406208 19517 leader_election.cc:302] T 1b9da24397c1439fbf4d7ecd26ff89f5 P d34bda967218462d8e840b14cd36cd95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e1a6a35fa24463391f1208fbd73e35c, d34bda967218462d8e840b14cd36cd95; no voters:
I20240503 14:13:07.406567 19680 raft_consensus.cc:2781] T 1b9da24397c1439fbf4d7ecd26ff89f5 P d34bda967218462d8e840b14cd36cd95 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:07.406896 19680 raft_consensus.cc:686] T 1b9da24397c1439fbf4d7ecd26ff89f5 P d34bda967218462d8e840b14cd36cd95 [term 1 LEADER]: Becoming Leader. State: Replica: d34bda967218462d8e840b14cd36cd95, State: Running, Role: LEADER
I20240503 14:13:07.407101 19680 consensus_queue.cc:237] T 1b9da24397c1439fbf4d7ecd26ff89f5 P d34bda967218462d8e840b14cd36cd95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e92d07359fbb434bbe6b3db778ad74be" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 45431 } } peers { permanent_uuid: "d34bda967218462d8e840b14cd36cd95" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39493 } } peers { permanent_uuid: "0e1a6a35fa24463391f1208fbd73e35c" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 45249 } }
I20240503 14:13:07.409224 19303 catalog_manager.cc:5484] T 1b9da24397c1439fbf4d7ecd26ff89f5 P d34bda967218462d8e840b14cd36cd95 reported cstate change: term changed from 0 to 1, leader changed from <none> to d34bda967218462d8e840b14cd36cd95 (127.16.94.131). New cstate: current_term: 1 leader_uuid: "d34bda967218462d8e840b14cd36cd95" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e92d07359fbb434bbe6b3db778ad74be" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 45431 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d34bda967218462d8e840b14cd36cd95" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39493 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e1a6a35fa24463391f1208fbd73e35c" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 45249 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:07.872217 19356 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:13:07.872627 19356 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:13:07.872730 19356 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:13:07.961351 19687 consensus_queue.cc:1035] T 1b9da24397c1439fbf4d7ecd26ff89f5 P d34bda967218462d8e840b14cd36cd95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e1a6a35fa24463391f1208fbd73e35c" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 45249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:07.965531 19690 consensus_queue.cc:1035] T 1b9da24397c1439fbf4d7ecd26ff89f5 P d34bda967218462d8e840b14cd36cd95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e92d07359fbb434bbe6b3db778ad74be" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 45431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:13:09.873075 19356 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:13:09.873521 19356 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:13:09.873615 19356 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:13:11.874051 19356 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:13:11.874470 19356 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:13:11.874548 19356 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:13:13.874915 19356 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:13:13.875373 19356 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:13:13.875481 19356 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:13:15.875840 19356 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:13:15.876288 19356 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:13:15.876374 19356 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:13:17.876721 19356 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:13:17.877208 19356 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:13:17.877313 19356 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:13:18.517194 16762 tablet_server.cc:178] TabletServer@127.16.94.129:0 shutting down...
I20240503 14:13:18.531291 16762 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 14:13:18.531709 16762 tablet_replica.cc:330] T 1b9da24397c1439fbf4d7ecd26ff89f5 P 0e1a6a35fa24463391f1208fbd73e35c: stopping tablet replica
I20240503 14:13:18.531997 16762 raft_consensus.cc:2229] T 1b9da24397c1439fbf4d7ecd26ff89f5 P 0e1a6a35fa24463391f1208fbd73e35c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:18.532290 16762 raft_consensus.cc:2258] T 1b9da24397c1439fbf4d7ecd26ff89f5 P 0e1a6a35fa24463391f1208fbd73e35c [term 1 FOLLOWER]: Raft consensus is shut down!
W20240503 14:13:18.545192 19517 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.94.129:45249: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20240503 14:13:18.546824 19517 consensus_peers.cc:485] T 1b9da24397c1439fbf4d7ecd26ff89f5 P d34bda967218462d8e840b14cd36cd95 -> Peer 0e1a6a35fa24463391f1208fbd73e35c (127.16.94.129:45249): Couldn't send request to peer 0e1a6a35fa24463391f1208fbd73e35c. Status: Network error: Client connection negotiation failed: client connection to 127.16.94.129:45249: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:18.547494 16762 tablet_server.cc:194] TabletServer@127.16.94.129:0 shutdown complete.
I20240503 14:13:18.550470 16762 tablet_server.cc:178] TabletServer@127.16.94.130:0 shutting down...
I20240503 14:13:18.561906 16762 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 14:13:18.562237 16762 tablet_replica.cc:330] T 1b9da24397c1439fbf4d7ecd26ff89f5 P e92d07359fbb434bbe6b3db778ad74be: stopping tablet replica
I20240503 14:13:18.562525 16762 raft_consensus.cc:2229] T 1b9da24397c1439fbf4d7ecd26ff89f5 P e92d07359fbb434bbe6b3db778ad74be [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:18.562772 16762 raft_consensus.cc:2258] T 1b9da24397c1439fbf4d7ecd26ff89f5 P e92d07359fbb434bbe6b3db778ad74be [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:18.577145 16762 tablet_server.cc:194] TabletServer@127.16.94.130:0 shutdown complete.
I20240503 14:13:18.580344 16762 tablet_server.cc:178] TabletServer@127.16.94.131:0 shutting down...
I20240503 14:13:18.591512 16762 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 14:13:18.591828 16762 tablet_replica.cc:330] T 1b9da24397c1439fbf4d7ecd26ff89f5 P d34bda967218462d8e840b14cd36cd95: stopping tablet replica
I20240503 14:13:18.592034 16762 raft_consensus.cc:2229] T 1b9da24397c1439fbf4d7ecd26ff89f5 P d34bda967218462d8e840b14cd36cd95 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:18.592392 16762 raft_consensus.cc:2258] T 1b9da24397c1439fbf4d7ecd26ff89f5 P d34bda967218462d8e840b14cd36cd95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:18.606359 16762 tablet_server.cc:194] TabletServer@127.16.94.131:0 shutdown complete.
I20240503 14:13:18.609566 16762 tablet_server.cc:178] TabletServer@127.16.94.132:0 shutting down...
I20240503 14:13:18.620353 16762 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 14:13:18.774369 16762 tablet_server.cc:194] TabletServer@127.16.94.132:0 shutdown complete.
I20240503 14:13:18.777179 16762 master.cc:537] Master@127.16.94.190:43231 shutting down...
I20240503 14:13:18.784888 16762 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 57be60b30b274fce8f0c2016ed5ff7ea [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:18.785149 16762 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 57be60b30b274fce8f0c2016ed5ff7ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:18.785272 16762 tablet_replica.cc:330] T 00000000000000000000000000000000 P 57be60b30b274fce8f0c2016ed5ff7ea: stopping tablet replica
I20240503 14:13:18.799669 16762 master.cc:552] Master@127.16.94.190:43231 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne (13004 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy
I20240503 14:13:18.816972 16762 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.16.94.190:38363
I20240503 14:13:18.817416 16762 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:13:18.820442 19704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:18.821062 19705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:18.821561 19707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:13:18.822104 16762 server_base.cc:1034] running on GCE node
I20240503 14:13:18.822305 16762 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:13:18.822355 16762 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:13:18.822396 16762 hybrid_clock.cc:648] HybridClock initialized: now 1714745598822392 us; error 0 us; skew 500 ppm
I20240503 14:13:18.822544 16762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:13:18.827694 16762 webserver.cc:460] Webserver started at http://127.16.94.190:37169/ using document root <none> and password file <none>
I20240503 14:13:18.827945 16762 fs_manager.cc:359] Metadata directory not provided
I20240503 14:13:18.828015 16762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:13:18.828140 16762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:13:18.828786 16762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714745515241994-16762-0/minicluster-data/master-0-root/instance:
uuid: "c75cf7fce5dd40ca8f332817edb124f2"
format_stamp: "Formatted at 2024-05-03 14:13:18 on dist-test-slave-v27g"
I20240503 14:13:18.831377 16762 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 14:13:18.832834 19712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:18.833130 16762 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 14:13:18.833225 16762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714745515241994-16762-0/minicluster-data/master-0-root
uuid: "c75cf7fce5dd40ca8f332817edb124f2"
format_stamp: "Formatted at 2024-05-03 14:13:18 on dist-test-slave-v27g"
I20240503 14:13:18.833323 16762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714745515241994-16762-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714745515241994-16762-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714745515241994-16762-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:13:18.845158 16762 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:13:18.845677 16762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:13:18.855618 16762 rpc_server.cc:307] RPC server started. Bound to: 127.16.94.190:38363
I20240503 14:13:18.855634 19763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.94.190:38363 every 8 connection(s)
I20240503 14:13:18.856806 19764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:18.860797 19764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c75cf7fce5dd40ca8f332817edb124f2: Bootstrap starting.
I20240503 14:13:18.861853 19764 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c75cf7fce5dd40ca8f332817edb124f2: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:18.863152 19764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c75cf7fce5dd40ca8f332817edb124f2: No bootstrap required, opened a new log
I20240503 14:13:18.863670 19764 raft_consensus.cc:348] T 00000000000000000000000000000000 P c75cf7fce5dd40ca8f332817edb124f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c75cf7fce5dd40ca8f332817edb124f2" member_type: VOTER }
I20240503 14:13:18.863788 19764 raft_consensus.cc:374] T 00000000000000000000000000000000 P c75cf7fce5dd40ca8f332817edb124f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:18.863842 19764 raft_consensus.cc:729] T 00000000000000000000000000000000 P c75cf7fce5dd40ca8f332817edb124f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c75cf7fce5dd40ca8f332817edb124f2, State: Initialized, Role: FOLLOWER
I20240503 14:13:18.864030 19764 consensus_queue.cc:260] T 00000000000000000000000000000000 P c75cf7fce5dd40ca8f332817edb124f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c75cf7fce5dd40ca8f332817edb124f2" member_type: VOTER }
I20240503 14:13:18.864133 19764 raft_consensus.cc:388] T 00000000000000000000000000000000 P c75cf7fce5dd40ca8f332817edb124f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:13:18.864189 19764 raft_consensus.cc:482] T 00000000000000000000000000000000 P c75cf7fce5dd40ca8f332817edb124f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:13:18.864257 19764 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c75cf7fce5dd40ca8f332817edb124f2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:18.865916 19764 raft_consensus.cc:504] T 00000000000000000000000000000000 P c75cf7fce5dd40ca8f332817edb124f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c75cf7fce5dd40ca8f332817edb124f2" member_type: VOTER }
I20240503 14:13:18.866079 19764 leader_election.cc:302] T 00000000000000000000000000000000 P c75cf7fce5dd40ca8f332817edb124f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c75cf7fce5dd40ca8f332817edb124f2; no voters:
I20240503 14:13:18.866401 19764 leader_election.cc:288] T 00000000000000000000000000000000 P c75cf7fce5dd40ca8f332817edb124f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:13:18.866487 19767 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c75cf7fce5dd40ca8f332817edb124f2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:18.866900 19767 raft_consensus.cc:686] T 00000000000000000000000000000000 P c75cf7fce5dd40ca8f332817edb124f2 [term 1 LEADER]: Becoming Leader. State: Replica: c75cf7fce5dd40ca8f332817edb124f2, State: Running, Role: LEADER
I20240503 14:13:18.867111 19767 consensus_queue.cc:237] T 00000000000000000000000000000000 P c75cf7fce5dd40ca8f332817edb124f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c75cf7fce5dd40ca8f332817edb124f2" member_type: VOTER }
I20240503 14:13:18.867175 19764 sys_catalog.cc:564] T 00000000000000000000000000000000 P c75cf7fce5dd40ca8f332817edb124f2 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 14:13:18.867755 19769 sys_catalog.cc:455] T 00000000000000000000000000000000 P c75cf7fce5dd40ca8f332817edb124f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c75cf7fce5dd40ca8f332817edb124f2. Latest consensus state: current_term: 1 leader_uuid: "c75cf7fce5dd40ca8f332817edb124f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c75cf7fce5dd40ca8f332817edb124f2" member_type: VOTER } }
I20240503 14:13:18.867717 19768 sys_catalog.cc:455] T 00000000000000000000000000000000 P c75cf7fce5dd40ca8f332817edb124f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c75cf7fce5dd40ca8f332817edb124f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c75cf7fce5dd40ca8f332817edb124f2" member_type: VOTER } }
I20240503 14:13:18.867879 19769 sys_catalog.cc:458] T 00000000000000000000000000000000 P c75cf7fce5dd40ca8f332817edb124f2 [sys.catalog]: This master's current role is: LEADER
I20240503 14:13:18.867899 19768 sys_catalog.cc:458] T 00000000000000000000000000000000 P c75cf7fce5dd40ca8f332817edb124f2 [sys.catalog]: This master's current role is: LEADER
I20240503 14:13:18.868824 19771 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 14:13:18.869959 19771 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 14:13:18.872730 19771 catalog_manager.cc:1347] Generated new cluster ID: e504b3593c3b4a00bef314fef0e9c702
I20240503 14:13:18.872800 19771 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 14:13:18.873921 16762 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 14:13:18.902107 19771 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 14:13:18.902848 19771 catalog_manager.cc:1504] Loading token signing keys...
I20240503 14:13:18.914942 19771 catalog_manager.cc:5857] T 00000000000000000000000000000000 P c75cf7fce5dd40ca8f332817edb124f2: Generated new TSK 0
I20240503 14:13:18.915186 19771 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 14:13:18.938736 16762 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:13:18.941800 19785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:18.942371 19786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:18.942992 19788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:13:18.943428 16762 server_base.cc:1034] running on GCE node
I20240503 14:13:18.943656 16762 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:13:18.943706 16762 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:13:18.943746 16762 hybrid_clock.cc:648] HybridClock initialized: now 1714745598943742 us; error 0 us; skew 500 ppm
I20240503 14:13:18.943894 16762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:13:18.945092 16762 webserver.cc:460] Webserver started at http://127.16.94.129:45963/ using document root <none> and password file <none>
I20240503 14:13:18.945297 16762 fs_manager.cc:359] Metadata directory not provided
I20240503 14:13:18.945358 16762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:13:18.945487 16762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:13:18.946166 16762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714745515241994-16762-0/minicluster-data/ts-0-root/instance:
uuid: "d95738ef1f1e4ac4bb4c7cde50078e17"
format_stamp: "Formatted at 2024-05-03 14:13:18 on dist-test-slave-v27g"
I20240503 14:13:18.948558 16762 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 14:13:18.950127 19793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:18.950495 16762 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 14:13:18.950577 16762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714745515241994-16762-0/minicluster-data/ts-0-root
uuid: "d95738ef1f1e4ac4bb4c7cde50078e17"
format_stamp: "Formatted at 2024-05-03 14:13:18 on dist-test-slave-v27g"
I20240503 14:13:18.950667 16762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714745515241994-16762-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714745515241994-16762-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714745515241994-16762-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:13:18.958971 16762 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:13:18.959403 16762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:13:18.959832 16762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:13:18.960628 16762 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:13:18.960678 16762 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:18.960742 16762 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:13:18.960772 16762 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:18.973503 16762 rpc_server.cc:307] RPC server started. Bound to: 127.16.94.129:37995
I20240503 14:13:18.973534 19857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.94.129:37995 every 8 connection(s)
I20240503 14:13:18.974972 16762 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:13:18.978386 19862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:18.978873 19863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:18.979352 19865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:13:18.979911 16762 server_base.cc:1034] running on GCE node
I20240503 14:13:18.980196 16762 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:13:18.980242 16762 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:13:18.980273 16762 hybrid_clock.cc:648] HybridClock initialized: now 1714745598980269 us; error 0 us; skew 500 ppm
I20240503 14:13:18.980382 16762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:13:18.981633 16762 webserver.cc:460] Webserver started at http://127.16.94.130:43217/ using document root <none> and password file <none>
I20240503 14:13:18.981879 16762 fs_manager.cc:359] Metadata directory not provided
I20240503 14:13:18.981961 16762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:13:18.982080 16762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:13:18.982092 19858 heartbeater.cc:347] Connected to a master server at 127.16.94.190:38363
I20240503 14:13:18.982215 19858 heartbeater.cc:464] Registering TS with master...
I20240503 14:13:18.982535 19858 heartbeater.cc:511] Master 127.16.94.190:38363 requested a full tablet report, sending...
I20240503 14:13:18.982702 16762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714745515241994-16762-0/minicluster-data/ts-1-root/instance:
uuid: "47ed5675049d492bb5db4be1bf60daa2"
format_stamp: "Formatted at 2024-05-03 14:13:18 on dist-test-slave-v27g"
I20240503 14:13:18.983377 19729 ts_manager.cc:194] Registered new tserver with Master: d95738ef1f1e4ac4bb4c7cde50078e17 (127.16.94.129:37995)
I20240503 14:13:18.984792 19729 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35402
I20240503 14:13:18.985726 16762 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20240503 14:13:18.987111 19870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:18.987412 16762 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240503 14:13:18.987490 16762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714745515241994-16762-0/minicluster-data/ts-1-root
uuid: "47ed5675049d492bb5db4be1bf60daa2"
format_stamp: "Formatted at 2024-05-03 14:13:18 on dist-test-slave-v27g"
I20240503 14:13:18.987591 16762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714745515241994-16762-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714745515241994-16762-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714745515241994-16762-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:13:18.998337 16762 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:13:18.998793 16762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:13:18.999248 16762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:13:18.999955 16762 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:13:19.000007 16762 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:19.000077 16762 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:13:19.000113 16762 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:19.012611 16762 rpc_server.cc:307] RPC server started. Bound to: 127.16.94.130:46867
I20240503 14:13:19.012639 19932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.94.130:46867 every 8 connection(s)
I20240503 14:13:19.014148 16762 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:13:19.017424 19936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:19.017896 19937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:19.018429 19939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:13:19.018962 16762 server_base.cc:1034] running on GCE node
I20240503 14:13:19.019449 16762 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:13:19.019544 16762 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:13:19.019615 16762 hybrid_clock.cc:648] HybridClock initialized: now 1714745599019588 us; error 0 us; skew 500 ppm
I20240503 14:13:19.019793 16762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:13:19.020519 19933 heartbeater.cc:347] Connected to a master server at 127.16.94.190:38363
I20240503 14:13:19.020681 19933 heartbeater.cc:464] Registering TS with master...
I20240503 14:13:19.021015 19933 heartbeater.cc:511] Master 127.16.94.190:38363 requested a full tablet report, sending...
I20240503 14:13:19.021222 16762 webserver.cc:460] Webserver started at http://127.16.94.131:45967/ using document root <none> and password file <none>
I20240503 14:13:19.021449 16762 fs_manager.cc:359] Metadata directory not provided
I20240503 14:13:19.021559 16762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:13:19.021729 16762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:13:19.021982 19729 ts_manager.cc:194] Registered new tserver with Master: 47ed5675049d492bb5db4be1bf60daa2 (127.16.94.130:46867)
I20240503 14:13:19.022509 16762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714745515241994-16762-0/minicluster-data/ts-2-root/instance:
uuid: "65d1bcf18f61420eb26158ea59b283eb"
format_stamp: "Formatted at 2024-05-03 14:13:19 on dist-test-slave-v27g"
I20240503 14:13:19.023099 19729 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35410
I20240503 14:13:19.025153 16762 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 14:13:19.026613 19945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:19.026926 16762 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240503 14:13:19.027040 16762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714745515241994-16762-0/minicluster-data/ts-2-root
uuid: "65d1bcf18f61420eb26158ea59b283eb"
format_stamp: "Formatted at 2024-05-03 14:13:19 on dist-test-slave-v27g"
I20240503 14:13:19.027153 16762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714745515241994-16762-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714745515241994-16762-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714745515241994-16762-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:13:19.036239 16762 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:13:19.036718 16762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:13:19.037202 16762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:13:19.038012 16762 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:13:19.038085 16762 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:19.038187 16762 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:13:19.038247 16762 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:19.048878 16762 rpc_server.cc:307] RPC server started. Bound to: 127.16.94.131:38497
I20240503 14:13:19.048905 20007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.94.131:38497 every 8 connection(s)
I20240503 14:13:19.055197 20008 heartbeater.cc:347] Connected to a master server at 127.16.94.190:38363
I20240503 14:13:19.055341 20008 heartbeater.cc:464] Registering TS with master...
I20240503 14:13:19.055683 20008 heartbeater.cc:511] Master 127.16.94.190:38363 requested a full tablet report, sending...
I20240503 14:13:19.056433 19729 ts_manager.cc:194] Registered new tserver with Master: 65d1bcf18f61420eb26158ea59b283eb (127.16.94.131:38497)
I20240503 14:13:19.056512 16762 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.006834973s
I20240503 14:13:19.057237 19729 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35418
I20240503 14:13:19.986192 19858 heartbeater.cc:503] Master 127.16.94.190:38363 was elected leader, sending a full tablet report...
I20240503 14:13:20.024340 19933 heartbeater.cc:503] Master 127.16.94.190:38363 was elected leader, sending a full tablet report...
I20240503 14:13:20.058553 20008 heartbeater.cc:503] Master 127.16.94.190:38363 was elected leader, sending a full tablet report...
I20240503 14:13:20.090050 16762 test_util.cc:268] Using random seed: 1381998511
I20240503 14:13:20.104089 19729 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35438:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:13:20.104694 19729 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:13:20.131556 19898 tablet_service.cc:1467] Processing CreateTablet for tablet 0e07322007924d8699e150a5605507ef (DEFAULT_TABLE table=test-workload [id=df627119575242adab424c677d2805cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:20.132099 19898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e07322007924d8699e150a5605507ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:20.133543 19973 tablet_service.cc:1467] Processing CreateTablet for tablet 0e07322007924d8699e150a5605507ef (DEFAULT_TABLE table=test-workload [id=df627119575242adab424c677d2805cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:20.133965 19973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e07322007924d8699e150a5605507ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:20.134625 19821 tablet_service.cc:1467] Processing CreateTablet for tablet 0e07322007924d8699e150a5605507ef (DEFAULT_TABLE table=test-workload [id=df627119575242adab424c677d2805cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:20.135018 19821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e07322007924d8699e150a5605507ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:20.137746 20029 tablet_bootstrap.cc:492] T 0e07322007924d8699e150a5605507ef P 47ed5675049d492bb5db4be1bf60daa2: Bootstrap starting.
I20240503 14:13:20.139098 20029 tablet_bootstrap.cc:654] T 0e07322007924d8699e150a5605507ef P 47ed5675049d492bb5db4be1bf60daa2: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:20.141145 20029 tablet_bootstrap.cc:492] T 0e07322007924d8699e150a5605507ef P 47ed5675049d492bb5db4be1bf60daa2: No bootstrap required, opened a new log
I20240503 14:13:20.141294 20029 ts_tablet_manager.cc:1367] T 0e07322007924d8699e150a5605507ef P 47ed5675049d492bb5db4be1bf60daa2: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:13:20.142100 20031 tablet_bootstrap.cc:492] T 0e07322007924d8699e150a5605507ef P 65d1bcf18f61420eb26158ea59b283eb: Bootstrap starting.
I20240503 14:13:20.142102 20029 raft_consensus.cc:348] T 0e07322007924d8699e150a5605507ef P 47ed5675049d492bb5db4be1bf60daa2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47ed5675049d492bb5db4be1bf60daa2" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 46867 } } peers { permanent_uuid: "65d1bcf18f61420eb26158ea59b283eb" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 38497 } } peers { permanent_uuid: "d95738ef1f1e4ac4bb4c7cde50078e17" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37995 } }
I20240503 14:13:20.142722 20029 raft_consensus.cc:374] T 0e07322007924d8699e150a5605507ef P 47ed5675049d492bb5db4be1bf60daa2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:20.142817 20029 raft_consensus.cc:729] T 0e07322007924d8699e150a5605507ef P 47ed5675049d492bb5db4be1bf60daa2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47ed5675049d492bb5db4be1bf60daa2, State: Initialized, Role: FOLLOWER
I20240503 14:13:20.143071 20029 consensus_queue.cc:260] T 0e07322007924d8699e150a5605507ef P 47ed5675049d492bb5db4be1bf60daa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47ed5675049d492bb5db4be1bf60daa2" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 46867 } } peers { permanent_uuid: "65d1bcf18f61420eb26158ea59b283eb" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 38497 } } peers { permanent_uuid: "d95738ef1f1e4ac4bb4c7cde50078e17" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37995 } }
I20240503 14:13:20.143537 20032 tablet_bootstrap.cc:492] T 0e07322007924d8699e150a5605507ef P d95738ef1f1e4ac4bb4c7cde50078e17: Bootstrap starting.
I20240503 14:13:20.143792 20031 tablet_bootstrap.cc:654] T 0e07322007924d8699e150a5605507ef P 65d1bcf18f61420eb26158ea59b283eb: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:20.145814 20029 ts_tablet_manager.cc:1398] T 0e07322007924d8699e150a5605507ef P 47ed5675049d492bb5db4be1bf60daa2: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:13:20.146382 20032 tablet_bootstrap.cc:654] T 0e07322007924d8699e150a5605507ef P d95738ef1f1e4ac4bb4c7cde50078e17: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:20.147353 20031 tablet_bootstrap.cc:492] T 0e07322007924d8699e150a5605507ef P 65d1bcf18f61420eb26158ea59b283eb: No bootstrap required, opened a new log
I20240503 14:13:20.147505 20031 ts_tablet_manager.cc:1367] T 0e07322007924d8699e150a5605507ef P 65d1bcf18f61420eb26158ea59b283eb: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20240503 14:13:20.148169 20032 tablet_bootstrap.cc:492] T 0e07322007924d8699e150a5605507ef P d95738ef1f1e4ac4bb4c7cde50078e17: No bootstrap required, opened a new log
I20240503 14:13:20.148356 20032 ts_tablet_manager.cc:1367] T 0e07322007924d8699e150a5605507ef P d95738ef1f1e4ac4bb4c7cde50078e17: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 14:13:20.148262 20031 raft_consensus.cc:348] T 0e07322007924d8699e150a5605507ef P 65d1bcf18f61420eb26158ea59b283eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47ed5675049d492bb5db4be1bf60daa2" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 46867 } } peers { permanent_uuid: "65d1bcf18f61420eb26158ea59b283eb" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 38497 } } peers { permanent_uuid: "d95738ef1f1e4ac4bb4c7cde50078e17" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37995 } }
I20240503 14:13:20.148464 20031 raft_consensus.cc:374] T 0e07322007924d8699e150a5605507ef P 65d1bcf18f61420eb26158ea59b283eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:20.148567 20031 raft_consensus.cc:729] T 0e07322007924d8699e150a5605507ef P 65d1bcf18f61420eb26158ea59b283eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65d1bcf18f61420eb26158ea59b283eb, State: Initialized, Role: FOLLOWER
I20240503 14:13:20.148820 20031 consensus_queue.cc:260] T 0e07322007924d8699e150a5605507ef P 65d1bcf18f61420eb26158ea59b283eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47ed5675049d492bb5db4be1bf60daa2" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 46867 } } peers { permanent_uuid: "65d1bcf18f61420eb26158ea59b283eb" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 38497 } } peers { permanent_uuid: "d95738ef1f1e4ac4bb4c7cde50078e17" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37995 } }
I20240503 14:13:20.149089 20032 raft_consensus.cc:348] T 0e07322007924d8699e150a5605507ef P d95738ef1f1e4ac4bb4c7cde50078e17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47ed5675049d492bb5db4be1bf60daa2" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 46867 } } peers { permanent_uuid: "65d1bcf18f61420eb26158ea59b283eb" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 38497 } } peers { permanent_uuid: "d95738ef1f1e4ac4bb4c7cde50078e17" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37995 } }
I20240503 14:13:20.149348 20032 raft_consensus.cc:374] T 0e07322007924d8699e150a5605507ef P d95738ef1f1e4ac4bb4c7cde50078e17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:20.149480 20032 raft_consensus.cc:729] T 0e07322007924d8699e150a5605507ef P d95738ef1f1e4ac4bb4c7cde50078e17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d95738ef1f1e4ac4bb4c7cde50078e17, State: Initialized, Role: FOLLOWER
I20240503 14:13:20.149685 20031 ts_tablet_manager.cc:1398] T 0e07322007924d8699e150a5605507ef P 65d1bcf18f61420eb26158ea59b283eb: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240503 14:13:20.149781 20032 consensus_queue.cc:260] T 0e07322007924d8699e150a5605507ef P d95738ef1f1e4ac4bb4c7cde50078e17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47ed5675049d492bb5db4be1bf60daa2" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 46867 } } peers { permanent_uuid: "65d1bcf18f61420eb26158ea59b283eb" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 38497 } } peers { permanent_uuid: "d95738ef1f1e4ac4bb4c7cde50078e17" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37995 } }
I20240503 14:13:20.150473 20032 ts_tablet_manager.cc:1398] T 0e07322007924d8699e150a5605507ef P d95738ef1f1e4ac4bb4c7cde50078e17: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:20.373991 20036 raft_consensus.cc:482] T 0e07322007924d8699e150a5605507ef P 65d1bcf18f61420eb26158ea59b283eb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:20.374190 20036 raft_consensus.cc:504] T 0e07322007924d8699e150a5605507ef P 65d1bcf18f61420eb26158ea59b283eb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47ed5675049d492bb5db4be1bf60daa2" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 46867 } } peers { permanent_uuid: "65d1bcf18f61420eb26158ea59b283eb" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 38497 } } peers { permanent_uuid: "d95738ef1f1e4ac4bb4c7cde50078e17" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37995 } }
I20240503 14:13:20.375021 20036 leader_election.cc:288] T 0e07322007924d8699e150a5605507ef P 65d1bcf18f61420eb26158ea59b283eb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47ed5675049d492bb5db4be1bf60daa2 (127.16.94.130:46867), d95738ef1f1e4ac4bb4c7cde50078e17 (127.16.94.129:37995)
I20240503 14:13:20.377810 20037 raft_consensus.cc:482] T 0e07322007924d8699e150a5605507ef P d95738ef1f1e4ac4bb4c7cde50078e17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:20.378010 20037 raft_consensus.cc:504] T 0e07322007924d8699e150a5605507ef P d95738ef1f1e4ac4bb4c7cde50078e17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47ed5675049d492bb5db4be1bf60daa2" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 46867 } } peers { permanent_uuid: "65d1bcf18f61420eb26158ea59b283eb" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 38497 } } peers { permanent_uuid: "d95738ef1f1e4ac4bb4c7cde50078e17" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37995 } }
I20240503 14:13:20.382375 19908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e07322007924d8699e150a5605507ef" candidate_uuid: "65d1bcf18f61420eb26158ea59b283eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47ed5675049d492bb5db4be1bf60daa2" is_pre_election: true
I20240503 14:13:20.382712 19908 raft_consensus.cc:2454] T 0e07322007924d8699e150a5605507ef P 47ed5675049d492bb5db4be1bf60daa2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65d1bcf18f61420eb26158ea59b283eb in term 0.
I20240503 14:13:20.382604 19831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e07322007924d8699e150a5605507ef" candidate_uuid: "65d1bcf18f61420eb26158ea59b283eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d95738ef1f1e4ac4bb4c7cde50078e17" is_pre_election: true
I20240503 14:13:20.382915 19831 raft_consensus.cc:2454] T 0e07322007924d8699e150a5605507ef P d95738ef1f1e4ac4bb4c7cde50078e17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65d1bcf18f61420eb26158ea59b283eb in term 0.
I20240503 14:13:20.383179 19948 leader_election.cc:302] T 0e07322007924d8699e150a5605507ef P 65d1bcf18f61420eb26158ea59b283eb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47ed5675049d492bb5db4be1bf60daa2, 65d1bcf18f61420eb26158ea59b283eb; no voters:
I20240503 14:13:20.383531 20036 raft_consensus.cc:2781] T 0e07322007924d8699e150a5605507ef P 65d1bcf18f61420eb26158ea59b283eb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:20.383643 20036 raft_consensus.cc:482] T 0e07322007924d8699e150a5605507ef P 65d1bcf18f61420eb26158ea59b283eb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:20.383694 20037 leader_election.cc:288] T 0e07322007924d8699e150a5605507ef P d95738ef1f1e4ac4bb4c7cde50078e17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47ed5675049d492bb5db4be1bf60daa2 (127.16.94.130:46867), 65d1bcf18f61420eb26158ea59b283eb (127.16.94.131:38497)
I20240503 14:13:20.383769 20036 raft_consensus.cc:3037] T 0e07322007924d8699e150a5605507ef P 65d1bcf18f61420eb26158ea59b283eb [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:20.386901 20036 raft_consensus.cc:504] T 0e07322007924d8699e150a5605507ef P 65d1bcf18f61420eb26158ea59b283eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47ed5675049d492bb5db4be1bf60daa2" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 46867 } } peers { permanent_uuid: "65d1bcf18f61420eb26158ea59b283eb" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 38497 } } peers { permanent_uuid: "d95738ef1f1e4ac4bb4c7cde50078e17" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37995 } }
I20240503 14:13:20.388165 19908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e07322007924d8699e150a5605507ef" candidate_uuid: "65d1bcf18f61420eb26158ea59b283eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47ed5675049d492bb5db4be1bf60daa2"
I20240503 14:13:20.388351 19908 raft_consensus.cc:3037] T 0e07322007924d8699e150a5605507ef P 47ed5675049d492bb5db4be1bf60daa2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:20.389092 19907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e07322007924d8699e150a5605507ef" candidate_uuid: "d95738ef1f1e4ac4bb4c7cde50078e17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47ed5675049d492bb5db4be1bf60daa2" is_pre_election: true
I20240503 14:13:20.389461 19831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e07322007924d8699e150a5605507ef" candidate_uuid: "65d1bcf18f61420eb26158ea59b283eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d95738ef1f1e4ac4bb4c7cde50078e17"
I20240503 14:13:20.389681 19831 raft_consensus.cc:3037] T 0e07322007924d8699e150a5605507ef P d95738ef1f1e4ac4bb4c7cde50078e17 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:20.390924 19908 raft_consensus.cc:2454] T 0e07322007924d8699e150a5605507ef P 47ed5675049d492bb5db4be1bf60daa2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65d1bcf18f61420eb26158ea59b283eb in term 1.
I20240503 14:13:20.391165 20036 leader_election.cc:288] T 0e07322007924d8699e150a5605507ef P 65d1bcf18f61420eb26158ea59b283eb [CANDIDATE]: Term 1 election: Requested vote from peers 47ed5675049d492bb5db4be1bf60daa2 (127.16.94.130:46867), d95738ef1f1e4ac4bb4c7cde50078e17 (127.16.94.129:37995)
I20240503 14:13:20.391273 19948 leader_election.cc:302] T 0e07322007924d8699e150a5605507ef P 65d1bcf18f61420eb26158ea59b283eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47ed5675049d492bb5db4be1bf60daa2, 65d1bcf18f61420eb26158ea59b283eb; no voters:
I20240503 14:13:20.391574 20036 raft_consensus.cc:2781] T 0e07322007924d8699e150a5605507ef P 65d1bcf18f61420eb26158ea59b283eb [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:20.392197 20036 raft_consensus.cc:686] T 0e07322007924d8699e150a5605507ef P 65d1bcf18f61420eb26158ea59b283eb [term 1 LEADER]: Becoming Leader. State: Replica: 65d1bcf18f61420eb26158ea59b283eb, State: Running, Role: LEADER
I20240503 14:13:20.392261 19831 raft_consensus.cc:2454] T 0e07322007924d8699e150a5605507ef P d95738ef1f1e4ac4bb4c7cde50078e17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65d1bcf18f61420eb26158ea59b283eb in term 1.
I20240503 14:13:20.392217 19983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e07322007924d8699e150a5605507ef" candidate_uuid: "d95738ef1f1e4ac4bb4c7cde50078e17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65d1bcf18f61420eb26158ea59b283eb" is_pre_election: true
I20240503 14:13:20.392819 20036 consensus_queue.cc:237] T 0e07322007924d8699e150a5605507ef P 65d1bcf18f61420eb26158ea59b283eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47ed5675049d492bb5db4be1bf60daa2" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 46867 } } peers { permanent_uuid: "65d1bcf18f61420eb26158ea59b283eb" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 38497 } } peers { permanent_uuid: "d95738ef1f1e4ac4bb4c7cde50078e17" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37995 } }
I20240503 14:13:20.392880 19795 leader_election.cc:302] T 0e07322007924d8699e150a5605507ef P d95738ef1f1e4ac4bb4c7cde50078e17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d95738ef1f1e4ac4bb4c7cde50078e17; no voters: 47ed5675049d492bb5db4be1bf60daa2, 65d1bcf18f61420eb26158ea59b283eb
I20240503 14:13:20.393131 20037 raft_consensus.cc:2726] T 0e07322007924d8699e150a5605507ef P d95738ef1f1e4ac4bb4c7cde50078e17 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:13:20.395325 19728 catalog_manager.cc:5484] T 0e07322007924d8699e150a5605507ef P 65d1bcf18f61420eb26158ea59b283eb reported cstate change: term changed from 0 to 1, leader changed from <none> to 65d1bcf18f61420eb26158ea59b283eb (127.16.94.131). New cstate: current_term: 1 leader_uuid: "65d1bcf18f61420eb26158ea59b283eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47ed5675049d492bb5db4be1bf60daa2" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 46867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65d1bcf18f61420eb26158ea59b283eb" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 38497 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d95738ef1f1e4ac4bb4c7cde50078e17" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37995 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:20.846925 20036 consensus_queue.cc:1035] T 0e07322007924d8699e150a5605507ef P 65d1bcf18f61420eb26158ea59b283eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d95738ef1f1e4ac4bb4c7cde50078e17" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:20.851248 20047 consensus_queue.cc:1035] T 0e07322007924d8699e150a5605507ef P 65d1bcf18f61420eb26158ea59b283eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "47ed5675049d492bb5db4be1bf60daa2" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 46867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:20.872502 19777 placement_policy_util.cc:407] tablet 0e07322007924d8699e150a5605507ef: detected majority of replicas (2 of 3) at location L0
I20240503 14:13:20.872866 19783 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:13:20.873267 19783 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:13:20.873365 19783 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:13:21.875229 19777 placement_policy_util.cc:407] tablet 0e07322007924d8699e150a5605507ef: detected majority of replicas (2 of 3) at location L0
I20240503 14:13:22.873755 19783 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:13:22.874214 19783 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:13:22.874316 19783 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:13:22.877470 19777 placement_policy_util.cc:407] tablet 0e07322007924d8699e150a5605507ef: detected majority of replicas (2 of 3) at location L0
I20240503 14:13:23.879812 19777 placement_policy_util.cc:407] tablet 0e07322007924d8699e150a5605507ef: detected majority of replicas (2 of 3) at location L0
I20240503 14:13:24.874752 19783 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:13:24.875203 19783 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:13:24.875308 19783 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:13:24.882196 19777 placement_policy_util.cc:407] tablet 0e07322007924d8699e150a5605507ef: detected majority of replicas (2 of 3) at location L0
I20240503 14:13:25.884546 19777 placement_policy_util.cc:407] tablet 0e07322007924d8699e150a5605507ef: detected majority of replicas (2 of 3) at location L0
I20240503 14:13:26.875659 19783 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:13:26.876096 19783 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:13:26.876209 19783 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:13:26.886984 19777 placement_policy_util.cc:407] tablet 0e07322007924d8699e150a5605507ef: detected majority of replicas (2 of 3) at location L0
I20240503 14:13:27.889426 19777 placement_policy_util.cc:407] tablet 0e07322007924d8699e150a5605507ef: detected majority of replicas (2 of 3) at location L0
I20240503 14:13:28.876706 19783 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:13:28.877147 19783 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:13:28.877255 19783 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:13:28.891891 19777 placement_policy_util.cc:407] tablet 0e07322007924d8699e150a5605507ef: detected majority of replicas (2 of 3) at location L0
I20240503 14:13:29.894275 19777 placement_policy_util.cc:407] tablet 0e07322007924d8699e150a5605507ef: detected majority of replicas (2 of 3) at location L0
I20240503 14:13:30.877585 19783 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:13:30.878041 19783 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:13:30.878144 19783 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:13:30.896668 19777 placement_policy_util.cc:407] tablet 0e07322007924d8699e150a5605507ef: detected majority of replicas (2 of 3) at location L0
I20240503 14:13:31.474552 16762 tablet_server.cc:178] TabletServer@127.16.94.129:0 shutting down...
I20240503 14:13:31.483863 16762 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 14:13:31.484126 16762 tablet_replica.cc:330] T 0e07322007924d8699e150a5605507ef P d95738ef1f1e4ac4bb4c7cde50078e17: stopping tablet replica
I20240503 14:13:31.484297 16762 raft_consensus.cc:2229] T 0e07322007924d8699e150a5605507ef P d95738ef1f1e4ac4bb4c7cde50078e17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:31.484493 16762 raft_consensus.cc:2258] T 0e07322007924d8699e150a5605507ef P d95738ef1f1e4ac4bb4c7cde50078e17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:31.505035 16762 tablet_server.cc:194] TabletServer@127.16.94.129:0 shutdown complete.
I20240503 14:13:31.507864 16762 tablet_server.cc:178] TabletServer@127.16.94.130:0 shutting down...
I20240503 14:13:31.516052 16762 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 14:13:31.516325 16762 tablet_replica.cc:330] T 0e07322007924d8699e150a5605507ef P 47ed5675049d492bb5db4be1bf60daa2: stopping tablet replica
I20240503 14:13:31.516510 16762 raft_consensus.cc:2229] T 0e07322007924d8699e150a5605507ef P 47ed5675049d492bb5db4be1bf60daa2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:31.516690 16762 raft_consensus.cc:2258] T 0e07322007924d8699e150a5605507ef P 47ed5675049d492bb5db4be1bf60daa2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:31.529944 16762 tablet_server.cc:194] TabletServer@127.16.94.130:0 shutdown complete.
I20240503 14:13:31.532615 16762 tablet_server.cc:178] TabletServer@127.16.94.131:0 shutting down...
I20240503 14:13:31.540518 16762 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 14:13:31.540731 16762 tablet_replica.cc:330] T 0e07322007924d8699e150a5605507ef P 65d1bcf18f61420eb26158ea59b283eb: stopping tablet replica
I20240503 14:13:31.540887 16762 raft_consensus.cc:2229] T 0e07322007924d8699e150a5605507ef P 65d1bcf18f61420eb26158ea59b283eb [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:31.541177 16762 raft_consensus.cc:2258] T 0e07322007924d8699e150a5605507ef P 65d1bcf18f61420eb26158ea59b283eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:31.554477 16762 tablet_server.cc:194] TabletServer@127.16.94.131:0 shutdown complete.
I20240503 14:13:31.557157 16762 master.cc:537] Master@127.16.94.190:38363 shutting down...
I20240503 14:13:31.564936 16762 raft_consensus.cc:2229] T 00000000000000000000000000000000 P c75cf7fce5dd40ca8f332817edb124f2 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:31.565166 16762 raft_consensus.cc:2258] T 00000000000000000000000000000000 P c75cf7fce5dd40ca8f332817edb124f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:31.565265 16762 tablet_replica.cc:330] T 00000000000000000000000000000000 P c75cf7fce5dd40ca8f332817edb124f2: stopping tablet replica
I20240503 14:13:31.578274 16762 master.cc:552] Master@127.16.94.190:38363 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy (12771 ms)
[ RUN ] AutoRebalancerTest.TestMaxMovesPerServer
I20240503 14:13:31.588797 16762 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.16.94.190:39321
I20240503 14:13:31.589298 16762 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:13:31.592253 20057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:31.592769 20058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:31.593288 20060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:13:31.593828 16762 server_base.cc:1034] running on GCE node
I20240503 14:13:31.594043 16762 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:13:31.594091 16762 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:13:31.594125 16762 hybrid_clock.cc:648] HybridClock initialized: now 1714745611594122 us; error 0 us; skew 500 ppm
I20240503 14:13:31.594276 16762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:13:31.595451 16762 webserver.cc:460] Webserver started at http://127.16.94.190:34203/ using document root <none> and password file <none>
I20240503 14:13:31.595657 16762 fs_manager.cc:359] Metadata directory not provided
I20240503 14:13:31.595715 16762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:13:31.595837 16762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:13:31.596565 16762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714745515241994-16762-0/minicluster-data/master-0-root/instance:
uuid: "ee3979ed1f72468fa7d996c2541237dc"
format_stamp: "Formatted at 2024-05-03 14:13:31 on dist-test-slave-v27g"
I20240503 14:13:31.598919 16762 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 14:13:31.600363 20065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:31.600698 16762 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 14:13:31.600793 16762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714745515241994-16762-0/minicluster-data/master-0-root
uuid: "ee3979ed1f72468fa7d996c2541237dc"
format_stamp: "Formatted at 2024-05-03 14:13:31 on dist-test-slave-v27g"
I20240503 14:13:31.600886 16762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714745515241994-16762-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714745515241994-16762-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714745515241994-16762-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:13:31.621294 16762 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:13:31.621803 16762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:13:31.631595 16762 rpc_server.cc:307] RPC server started. Bound to: 127.16.94.190:39321
I20240503 14:13:31.631615 20117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.94.190:39321 every 8 connection(s)
I20240503 14:13:31.632755 20118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:31.636582 20118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee3979ed1f72468fa7d996c2541237dc: Bootstrap starting.
I20240503 14:13:31.637552 20118 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee3979ed1f72468fa7d996c2541237dc: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:31.638803 20118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee3979ed1f72468fa7d996c2541237dc: No bootstrap required, opened a new log
I20240503 14:13:31.639282 20118 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee3979ed1f72468fa7d996c2541237dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee3979ed1f72468fa7d996c2541237dc" member_type: VOTER }
I20240503 14:13:31.639392 20118 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee3979ed1f72468fa7d996c2541237dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:31.639431 20118 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee3979ed1f72468fa7d996c2541237dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee3979ed1f72468fa7d996c2541237dc, State: Initialized, Role: FOLLOWER
I20240503 14:13:31.639585 20118 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee3979ed1f72468fa7d996c2541237dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee3979ed1f72468fa7d996c2541237dc" member_type: VOTER }
I20240503 14:13:31.639698 20118 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee3979ed1f72468fa7d996c2541237dc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:13:31.639739 20118 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee3979ed1f72468fa7d996c2541237dc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:13:31.639786 20118 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee3979ed1f72468fa7d996c2541237dc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:31.641371 20118 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee3979ed1f72468fa7d996c2541237dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee3979ed1f72468fa7d996c2541237dc" member_type: VOTER }
I20240503 14:13:31.641537 20118 leader_election.cc:302] T 00000000000000000000000000000000 P ee3979ed1f72468fa7d996c2541237dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee3979ed1f72468fa7d996c2541237dc; no voters:
I20240503 14:13:31.641878 20118 leader_election.cc:288] T 00000000000000000000000000000000 P ee3979ed1f72468fa7d996c2541237dc [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:13:31.641983 20121 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee3979ed1f72468fa7d996c2541237dc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:31.642468 20121 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee3979ed1f72468fa7d996c2541237dc [term 1 LEADER]: Becoming Leader. State: Replica: ee3979ed1f72468fa7d996c2541237dc, State: Running, Role: LEADER
I20240503 14:13:31.642668 20118 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee3979ed1f72468fa7d996c2541237dc [sys.catalog]: configured and running, proceeding with master startup.
I20240503 14:13:31.642675 20121 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee3979ed1f72468fa7d996c2541237dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee3979ed1f72468fa7d996c2541237dc" member_type: VOTER }
I20240503 14:13:31.643775 20122 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee3979ed1f72468fa7d996c2541237dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee3979ed1f72468fa7d996c2541237dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee3979ed1f72468fa7d996c2541237dc" member_type: VOTER } }
I20240503 14:13:31.643815 20123 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee3979ed1f72468fa7d996c2541237dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee3979ed1f72468fa7d996c2541237dc. Latest consensus state: current_term: 1 leader_uuid: "ee3979ed1f72468fa7d996c2541237dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee3979ed1f72468fa7d996c2541237dc" member_type: VOTER } }
I20240503 14:13:31.644054 20122 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee3979ed1f72468fa7d996c2541237dc [sys.catalog]: This master's current role is: LEADER
I20240503 14:13:31.644088 20123 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee3979ed1f72468fa7d996c2541237dc [sys.catalog]: This master's current role is: LEADER
I20240503 14:13:31.645141 20127 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 14:13:31.646404 20127 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 14:13:31.647611 16762 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 14:13:31.648936 20127 catalog_manager.cc:1347] Generated new cluster ID: e0cc7eb5d7da40b3ac3f6c7819ef53f6
I20240503 14:13:31.649065 20127 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 14:13:31.672780 20127 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 14:13:31.673480 20127 catalog_manager.cc:1504] Loading token signing keys...
I20240503 14:13:31.685492 20127 catalog_manager.cc:5857] T 00000000000000000000000000000000 P ee3979ed1f72468fa7d996c2541237dc: Generated new TSK 0
I20240503 14:13:31.685734 20127 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 14:13:31.712517 16762 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:13:31.715642 20140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:31.716150 20139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:31.716643 20142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:13:31.717267 16762 server_base.cc:1034] running on GCE node
I20240503 14:13:31.717499 16762 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:13:31.717547 16762 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:13:31.717571 16762 hybrid_clock.cc:648] HybridClock initialized: now 1714745611717566 us; error 0 us; skew 500 ppm
I20240503 14:13:31.717761 16762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:13:31.718962 16762 webserver.cc:460] Webserver started at http://127.16.94.129:36239/ using document root <none> and password file <none>
I20240503 14:13:31.719182 16762 fs_manager.cc:359] Metadata directory not provided
I20240503 14:13:31.719237 16762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:13:31.719363 16762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:13:31.719938 16762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714745515241994-16762-0/minicluster-data/ts-0-root/instance:
uuid: "64ad8799c8df4b7589869baffb93b68d"
format_stamp: "Formatted at 2024-05-03 14:13:31 on dist-test-slave-v27g"
I20240503 14:13:31.722355 16762 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 14:13:31.723771 20147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:31.724099 16762 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 14:13:31.724187 16762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714745515241994-16762-0/minicluster-data/ts-0-root
uuid: "64ad8799c8df4b7589869baffb93b68d"
format_stamp: "Formatted at 2024-05-03 14:13:31 on dist-test-slave-v27g"
I20240503 14:13:31.724288 16762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714745515241994-16762-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714745515241994-16762-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714745515241994-16762-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:13:31.752696 16762 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:13:31.753216 16762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:13:31.753791 16762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:13:31.754616 16762 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:13:31.754666 16762 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:31.754735 16762 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:13:31.754768 16762 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:31.766474 16762 rpc_server.cc:307] RPC server started. Bound to: 127.16.94.129:37429
I20240503 14:13:31.766487 20210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.94.129:37429 every 8 connection(s)
I20240503 14:13:31.767890 16762 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:13:31.771072 20215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:31.771631 20216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:31.772181 20218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:13:31.772675 16762 server_base.cc:1034] running on GCE node
I20240503 14:13:31.773023 16762 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:13:31.773123 16762 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:13:31.773173 16762 hybrid_clock.cc:648] HybridClock initialized: now 1714745611773170 us; error 0 us; skew 500 ppm
I20240503 14:13:31.773316 16762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:13:31.774703 20211 heartbeater.cc:347] Connected to a master server at 127.16.94.190:39321
I20240503 14:13:31.774705 16762 webserver.cc:460] Webserver started at http://127.16.94.130:41855/ using document root <none> and password file <none>
I20240503 14:13:31.774854 20211 heartbeater.cc:464] Registering TS with master...
I20240503 14:13:31.774961 16762 fs_manager.cc:359] Metadata directory not provided
I20240503 14:13:31.775033 16762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:13:31.775161 16762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:13:31.775231 20211 heartbeater.cc:511] Master 127.16.94.190:39321 requested a full tablet report, sending...
I20240503 14:13:31.775949 16762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714745515241994-16762-0/minicluster-data/ts-1-root/instance:
uuid: "a5d66df07715448fab64928f5bc8299a"
format_stamp: "Formatted at 2024-05-03 14:13:31 on dist-test-slave-v27g"
I20240503 14:13:31.776114 20083 ts_manager.cc:194] Registered new tserver with Master: 64ad8799c8df4b7589869baffb93b68d (127.16.94.129:37429)
I20240503 14:13:31.777041 20083 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51660
I20240503 14:13:31.778643 16762 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20240503 14:13:31.780126 20223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:31.780417 16762 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240503 14:13:31.780504 16762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714745515241994-16762-0/minicluster-data/ts-1-root
uuid: "a5d66df07715448fab64928f5bc8299a"
format_stamp: "Formatted at 2024-05-03 14:13:31 on dist-test-slave-v27g"
I20240503 14:13:31.780617 16762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714745515241994-16762-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714745515241994-16762-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714745515241994-16762-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:13:31.794590 16762 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:13:31.795053 16762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:13:31.795536 16762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:13:31.796291 16762 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:13:31.796343 16762 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:31.796412 16762 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:13:31.796447 16762 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:31.808079 16762 rpc_server.cc:307] RPC server started. Bound to: 127.16.94.130:39295
I20240503 14:13:31.808108 20285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.94.130:39295 every 8 connection(s)
I20240503 14:13:31.809458 16762 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:13:31.812438 20289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:31.813019 20290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:31.813423 20292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:13:31.813963 16762 server_base.cc:1034] running on GCE node
I20240503 14:13:31.814239 16762 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:13:31.814318 16762 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:13:31.814364 16762 hybrid_clock.cc:648] HybridClock initialized: now 1714745611814360 us; error 0 us; skew 500 ppm
I20240503 14:13:31.814491 16762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:13:31.815454 20286 heartbeater.cc:347] Connected to a master server at 127.16.94.190:39321
I20240503 14:13:31.815560 20286 heartbeater.cc:464] Registering TS with master...
I20240503 14:13:31.815891 16762 webserver.cc:460] Webserver started at http://127.16.94.131:35123/ using document root <none> and password file <none>
I20240503 14:13:31.815944 20286 heartbeater.cc:511] Master 127.16.94.190:39321 requested a full tablet report, sending...
I20240503 14:13:31.816097 16762 fs_manager.cc:359] Metadata directory not provided
I20240503 14:13:31.816159 16762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:13:31.816267 16762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:13:31.816742 20083 ts_manager.cc:194] Registered new tserver with Master: a5d66df07715448fab64928f5bc8299a (127.16.94.130:39295)
I20240503 14:13:31.816893 16762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714745515241994-16762-0/minicluster-data/ts-2-root/instance:
uuid: "fceaf96557de46c3986c306fa4ac3b9e"
format_stamp: "Formatted at 2024-05-03 14:13:31 on dist-test-slave-v27g"
I20240503 14:13:31.817430 20083 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51674
I20240503 14:13:31.819281 16762 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 14:13:31.820731 20297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:31.821033 16762 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 14:13:31.821123 16762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714745515241994-16762-0/minicluster-data/ts-2-root
uuid: "fceaf96557de46c3986c306fa4ac3b9e"
format_stamp: "Formatted at 2024-05-03 14:13:31 on dist-test-slave-v27g"
I20240503 14:13:31.821213 16762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714745515241994-16762-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714745515241994-16762-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714745515241994-16762-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:13:31.828723 16762 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:13:31.829178 16762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:13:31.829684 16762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:13:31.830448 16762 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:13:31.830502 16762 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:31.830567 16762 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:13:31.830602 16762 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:31.842723 16762 rpc_server.cc:307] RPC server started. Bound to: 127.16.94.131:39581
I20240503 14:13:31.842752 20359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.94.131:39581 every 8 connection(s)
I20240503 14:13:31.848642 20360 heartbeater.cc:347] Connected to a master server at 127.16.94.190:39321
I20240503 14:13:31.848760 20360 heartbeater.cc:464] Registering TS with master...
I20240503 14:13:31.849043 20360 heartbeater.cc:511] Master 127.16.94.190:39321 requested a full tablet report, sending...
I20240503 14:13:31.849786 20083 ts_manager.cc:194] Registered new tserver with Master: fceaf96557de46c3986c306fa4ac3b9e (127.16.94.131:39581)
I20240503 14:13:31.850396 16762 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.006781134s
I20240503 14:13:31.850472 20083 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51676
I20240503 14:13:32.778483 20211 heartbeater.cc:503] Master 127.16.94.190:39321 was elected leader, sending a full tablet report...
I20240503 14:13:32.818548 20286 heartbeater.cc:503] Master 127.16.94.190:39321 was elected leader, sending a full tablet report...
I20240503 14:13:32.851734 20360 heartbeater.cc:503] Master 127.16.94.190:39321 was elected leader, sending a full tablet report...
I20240503 14:13:32.877702 16762 test_util.cc:268] Using random seed: 1394786173
I20240503 14:13:32.890528 20083 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58834:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\252\252\252\n\004\001\000TUU\025\004\001\000\376\377\377\037\004\001\000\250\252\252*\004\001\000RUU5\004\001\000\374\377\377?\004\001\000\246\252\252J\004\001\000PUUU\004\001\000\372\377\377_\004\001\000\244\252\252j\004\001\000NUUu""\004\001\000\252\252\252\n\004\001\000TUU\025\004\001\000\376\377\377\037\004\001\000\250\252\252*\004\001\000RUU5\004\001\000\374\377\377?\004\001\000\246\252\252J\004\001\000PUUU\004\001\000\372\377\377_\004\001\000\244\252\252j\004\001\000NUUu"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:13:32.891323 20083 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:13:32.928278 20250 tablet_service.cc:1467] Processing CreateTablet for tablet 44a4d2b820fc4d0da06c659fe98fe57c (DEFAULT_TABLE table=test-workload [id=cc8b29b2b6ee4a4eb4823287254752f5]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20240503 14:13:32.928545 20247 tablet_service.cc:1467] Processing CreateTablet for tablet c45ceab3a36d4225aed74654ed530c9d (DEFAULT_TABLE table=test-workload [id=cc8b29b2b6ee4a4eb4823287254752f5]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20240503 14:13:32.928815 20250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44a4d2b820fc4d0da06c659fe98fe57c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:32.929147 20245 tablet_service.cc:1467] Processing CreateTablet for tablet 5484fffa7f6b4e64ba4798e8816b6c61 (DEFAULT_TABLE table=test-workload [id=cc8b29b2b6ee4a4eb4823287254752f5]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20240503 14:13:32.929265 20251 tablet_service.cc:1467] Processing CreateTablet for tablet 220ef8bab4694ace9bdf0fce06226efa (DEFAULT_TABLE table=test-workload [id=cc8b29b2b6ee4a4eb4823287254752f5]), partition=RANGE (key) PARTITION VALUES < 178956970
I20240503 14:13:32.929546 20245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5484fffa7f6b4e64ba4798e8816b6c61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:32.929771 20251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 220ef8bab4694ace9bdf0fce06226efa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:32.928278 20248 tablet_service.cc:1467] Processing CreateTablet for tablet 29f4ee03917646f6a5374217bed7a1be (DEFAULT_TABLE table=test-workload [id=cc8b29b2b6ee4a4eb4823287254752f5]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20240503 14:13:32.930176 20248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29f4ee03917646f6a5374217bed7a1be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:32.930406 20242 tablet_service.cc:1467] Processing CreateTablet for tablet 0eb7552695304a1ea6f8e02f189f7295 (DEFAULT_TABLE table=test-workload [id=cc8b29b2b6ee4a4eb4823287254752f5]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20240503 14:13:32.930810 20242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0eb7552695304a1ea6f8e02f189f7295. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:32.933775 20247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c45ceab3a36d4225aed74654ed530c9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:32.933979 20243 tablet_service.cc:1467] Processing CreateTablet for tablet 7a82ae1f4c66455caea352a304af2b51 (DEFAULT_TABLE table=test-workload [id=cc8b29b2b6ee4a4eb4823287254752f5]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20240503 14:13:32.934346 20243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a82ae1f4c66455caea352a304af2b51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:32.936681 20251 tablet_service.cc:1467] Processing CreateTablet for tablet 7905fd813a2d4a8aa8b2ac1e752b37dc (DEFAULT_TABLE table=test-workload [id=cc8b29b2b6ee4a4eb4823287254752f5]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20240503 14:13:32.937070 20251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7905fd813a2d4a8aa8b2ac1e752b37dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:32.938069 20244 tablet_service.cc:1467] Processing CreateTablet for tablet 904450ab5e8b4250b7af18b98d2b0985 (DEFAULT_TABLE table=test-workload [id=cc8b29b2b6ee4a4eb4823287254752f5]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20240503 14:13:32.938477 20244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 904450ab5e8b4250b7af18b98d2b0985. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:32.942040 20246 tablet_service.cc:1467] Processing CreateTablet for tablet d4743d5a73c54086b277f4c4b1a248fd (DEFAULT_TABLE table=test-workload [id=cc8b29b2b6ee4a4eb4823287254752f5]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20240503 14:13:32.942445 20246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4743d5a73c54086b277f4c4b1a248fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:32.945250 20249 tablet_service.cc:1467] Processing CreateTablet for tablet ad711d78ee524d979867b65906abe456 (DEFAULT_TABLE table=test-workload [id=cc8b29b2b6ee4a4eb4823287254752f5]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20240503 14:13:32.945555 20249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad711d78ee524d979867b65906abe456. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:32.950301 20244 tablet_service.cc:1467] Processing CreateTablet for tablet 3054a11a8da4470ea5241b92d164d6a8 (DEFAULT_TABLE table=test-workload [id=cc8b29b2b6ee4a4eb4823287254752f5]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20240503 14:13:32.950691 20244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3054a11a8da4470ea5241b92d164d6a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:32.952119 20175 tablet_service.cc:1467] Processing CreateTablet for tablet 220ef8bab4694ace9bdf0fce06226efa (DEFAULT_TABLE table=test-workload [id=cc8b29b2b6ee4a4eb4823287254752f5]), partition=RANGE (key) PARTITION VALUES < 178956970
I20240503 14:13:32.952489 20175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 220ef8bab4694ace9bdf0fce06226efa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:32.954353 20380 tablet_bootstrap.cc:492] T 44a4d2b820fc4d0da06c659fe98fe57c P a5d66df07715448fab64928f5bc8299a: Bootstrap starting.
I20240503 14:13:32.955494 20380 tablet_bootstrap.cc:654] T 44a4d2b820fc4d0da06c659fe98fe57c P a5d66df07715448fab64928f5bc8299a: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:32.958196 20174 tablet_service.cc:1467] Processing CreateTablet for tablet 44a4d2b820fc4d0da06c659fe98fe57c (DEFAULT_TABLE table=test-workload [id=cc8b29b2b6ee4a4eb4823287254752f5]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20240503 14:13:32.958489 20380 tablet_bootstrap.cc:492] T 44a4d2b820fc4d0da06c659fe98fe57c P a5d66df07715448fab64928f5bc8299a: No bootstrap required, opened a new log
I20240503 14:13:32.958495 20174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44a4d2b820fc4d0da06c659fe98fe57c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:32.958637 20380 ts_tablet_manager.cc:1367] T 44a4d2b820fc4d0da06c659fe98fe57c P a5d66df07715448fab64928f5bc8299a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:13:32.958791 20173 tablet_service.cc:1467] Processing CreateTablet for tablet ad711d78ee524d979867b65906abe456 (DEFAULT_TABLE table=test-workload [id=cc8b29b2b6ee4a4eb4823287254752f5]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20240503 14:13:32.959144 20173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad711d78ee524d979867b65906abe456. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:32.959273 20380 raft_consensus.cc:348] T 44a4d2b820fc4d0da06c659fe98fe57c P a5d66df07715448fab64928f5bc8299a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:32.959492 20380 raft_consensus.cc:374] T 44a4d2b820fc4d0da06c659fe98fe57c P a5d66df07715448fab64928f5bc8299a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:32.959546 20380 raft_consensus.cc:729] T 44a4d2b820fc4d0da06c659fe98fe57c P a5d66df07715448fab64928f5bc8299a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5d66df07715448fab64928f5bc8299a, State: Initialized, Role: FOLLOWER
I20240503 14:13:32.959827 20380 consensus_queue.cc:260] T 44a4d2b820fc4d0da06c659fe98fe57c P a5d66df07715448fab64928f5bc8299a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:32.961490 20172 tablet_service.cc:1467] Processing CreateTablet for tablet 29f4ee03917646f6a5374217bed7a1be (DEFAULT_TABLE table=test-workload [id=cc8b29b2b6ee4a4eb4823287254752f5]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20240503 14:13:32.961916 20172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29f4ee03917646f6a5374217bed7a1be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:32.961522 20167 tablet_service.cc:1467] Processing CreateTablet for tablet 7a82ae1f4c66455caea352a304af2b51 (DEFAULT_TABLE table=test-workload [id=cc8b29b2b6ee4a4eb4823287254752f5]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20240503 14:13:32.961490 20166 tablet_service.cc:1467] Processing CreateTablet for tablet 0eb7552695304a1ea6f8e02f189f7295 (DEFAULT_TABLE table=test-workload [id=cc8b29b2b6ee4a4eb4823287254752f5]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20240503 14:13:32.964861 20169 tablet_service.cc:1467] Processing CreateTablet for tablet 5484fffa7f6b4e64ba4798e8816b6c61 (DEFAULT_TABLE table=test-workload [id=cc8b29b2b6ee4a4eb4823287254752f5]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20240503 14:13:32.965040 20166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0eb7552695304a1ea6f8e02f189f7295. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:32.965008 20171 tablet_service.cc:1467] Processing CreateTablet for tablet c45ceab3a36d4225aed74654ed530c9d (DEFAULT_TABLE table=test-workload [id=cc8b29b2b6ee4a4eb4823287254752f5]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20240503 14:13:32.965248 20169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5484fffa7f6b4e64ba4798e8816b6c61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:32.965461 20170 tablet_service.cc:1467] Processing CreateTablet for tablet d4743d5a73c54086b277f4c4b1a248fd (DEFAULT_TABLE table=test-workload [id=cc8b29b2b6ee4a4eb4823287254752f5]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20240503 14:13:32.965883 20170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4743d5a73c54086b277f4c4b1a248fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:32.970152 20325 tablet_service.cc:1467] Processing CreateTablet for tablet 220ef8bab4694ace9bdf0fce06226efa (DEFAULT_TABLE table=test-workload [id=cc8b29b2b6ee4a4eb4823287254752f5]), partition=RANGE (key) PARTITION VALUES < 178956970
I20240503 14:13:32.970515 20325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 220ef8bab4694ace9bdf0fce06226efa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:32.970837 20317 tablet_service.cc:1467] Processing CreateTablet for tablet 7a82ae1f4c66455caea352a304af2b51 (DEFAULT_TABLE table=test-workload [id=cc8b29b2b6ee4a4eb4823287254752f5]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20240503 14:13:32.971158 20167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a82ae1f4c66455caea352a304af2b51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:32.971416 20317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a82ae1f4c66455caea352a304af2b51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:32.971472 20324 tablet_service.cc:1467] Processing CreateTablet for tablet 44a4d2b820fc4d0da06c659fe98fe57c (DEFAULT_TABLE table=test-workload [id=cc8b29b2b6ee4a4eb4823287254752f5]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20240503 14:13:32.971879 20324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44a4d2b820fc4d0da06c659fe98fe57c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:32.972196 20174 tablet_service.cc:1467] Processing CreateTablet for tablet 3054a11a8da4470ea5241b92d164d6a8 (DEFAULT_TABLE table=test-workload [id=cc8b29b2b6ee4a4eb4823287254752f5]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20240503 14:13:32.972512 20174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3054a11a8da4470ea5241b92d164d6a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:32.973945 20323 tablet_service.cc:1467] Processing CreateTablet for tablet ad711d78ee524d979867b65906abe456 (DEFAULT_TABLE table=test-workload [id=cc8b29b2b6ee4a4eb4823287254752f5]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20240503 14:13:32.971393 20172 tablet_service.cc:1467] Processing CreateTablet for tablet 7905fd813a2d4a8aa8b2ac1e752b37dc (DEFAULT_TABLE table=test-workload [id=cc8b29b2b6ee4a4eb4823287254752f5]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20240503 14:13:32.974921 20171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c45ceab3a36d4225aed74654ed530c9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:32.975102 20172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7905fd813a2d4a8aa8b2ac1e752b37dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:32.975667 20323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad711d78ee524d979867b65906abe456. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:32.964869 20168 tablet_service.cc:1467] Processing CreateTablet for tablet 904450ab5e8b4250b7af18b98d2b0985 (DEFAULT_TABLE table=test-workload [id=cc8b29b2b6ee4a4eb4823287254752f5]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20240503 14:13:32.979377 20321 tablet_service.cc:1467] Processing CreateTablet for tablet c45ceab3a36d4225aed74654ed530c9d (DEFAULT_TABLE table=test-workload [id=cc8b29b2b6ee4a4eb4823287254752f5]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20240503 14:13:32.982184 20319 tablet_service.cc:1467] Processing CreateTablet for tablet 5484fffa7f6b4e64ba4798e8816b6c61 (DEFAULT_TABLE table=test-workload [id=cc8b29b2b6ee4a4eb4823287254752f5]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20240503 14:13:32.983302 20321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c45ceab3a36d4225aed74654ed530c9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:32.985987 20318 tablet_service.cc:1467] Processing CreateTablet for tablet 904450ab5e8b4250b7af18b98d2b0985 (DEFAULT_TABLE table=test-workload [id=cc8b29b2b6ee4a4eb4823287254752f5]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20240503 14:13:32.987730 20320 tablet_service.cc:1467] Processing CreateTablet for tablet d4743d5a73c54086b277f4c4b1a248fd (DEFAULT_TABLE table=test-workload [id=cc8b29b2b6ee4a4eb4823287254752f5]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20240503 14:13:32.986472 20316 tablet_service.cc:1467] Processing CreateTablet for tablet 0eb7552695304a1ea6f8e02f189f7295 (DEFAULT_TABLE table=test-workload [id=cc8b29b2b6ee4a4eb4823287254752f5]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20240503 14:13:32.991379 20322 tablet_service.cc:1467] Processing CreateTablet for tablet 29f4ee03917646f6a5374217bed7a1be (DEFAULT_TABLE table=test-workload [id=cc8b29b2b6ee4a4eb4823287254752f5]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20240503 14:13:32.993809 20168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 904450ab5e8b4250b7af18b98d2b0985. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:33.001837 20318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 904450ab5e8b4250b7af18b98d2b0985. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:33.009840 20320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4743d5a73c54086b277f4c4b1a248fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:33.011063 20316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0eb7552695304a1ea6f8e02f189f7295. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:33.014788 20380 ts_tablet_manager.cc:1398] T 44a4d2b820fc4d0da06c659fe98fe57c P a5d66df07715448fab64928f5bc8299a: Time spent starting tablet: real 0.056s user 0.006s sys 0.034s
I20240503 14:13:33.015125 20380 tablet_bootstrap.cc:492] T 220ef8bab4694ace9bdf0fce06226efa P a5d66df07715448fab64928f5bc8299a: Bootstrap starting.
I20240503 14:13:33.016471 20325 tablet_service.cc:1467] Processing CreateTablet for tablet 7905fd813a2d4a8aa8b2ac1e752b37dc (DEFAULT_TABLE table=test-workload [id=cc8b29b2b6ee4a4eb4823287254752f5]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20240503 14:13:33.016970 20325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7905fd813a2d4a8aa8b2ac1e752b37dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:33.019484 20319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5484fffa7f6b4e64ba4798e8816b6c61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:33.026774 20323 tablet_service.cc:1467] Processing CreateTablet for tablet 3054a11a8da4470ea5241b92d164d6a8 (DEFAULT_TABLE table=test-workload [id=cc8b29b2b6ee4a4eb4823287254752f5]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20240503 14:13:33.030232 20322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29f4ee03917646f6a5374217bed7a1be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:33.036957 20323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3054a11a8da4470ea5241b92d164d6a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:33.053491 20383 tablet_bootstrap.cc:492] T 220ef8bab4694ace9bdf0fce06226efa P 64ad8799c8df4b7589869baffb93b68d: Bootstrap starting.
I20240503 14:13:33.055526 20380 tablet_bootstrap.cc:654] T 220ef8bab4694ace9bdf0fce06226efa P a5d66df07715448fab64928f5bc8299a: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:33.058131 20384 tablet_bootstrap.cc:492] T 7a82ae1f4c66455caea352a304af2b51 P fceaf96557de46c3986c306fa4ac3b9e: Bootstrap starting.
I20240503 14:13:33.060777 20384 tablet_bootstrap.cc:654] T 7a82ae1f4c66455caea352a304af2b51 P fceaf96557de46c3986c306fa4ac3b9e: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:33.062561 20384 tablet_bootstrap.cc:492] T 7a82ae1f4c66455caea352a304af2b51 P fceaf96557de46c3986c306fa4ac3b9e: No bootstrap required, opened a new log
I20240503 14:13:33.062692 20384 ts_tablet_manager.cc:1367] T 7a82ae1f4c66455caea352a304af2b51 P fceaf96557de46c3986c306fa4ac3b9e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 14:13:33.062798 20383 tablet_bootstrap.cc:654] T 220ef8bab4694ace9bdf0fce06226efa P 64ad8799c8df4b7589869baffb93b68d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:33.063421 20384 raft_consensus.cc:348] T 7a82ae1f4c66455caea352a304af2b51 P fceaf96557de46c3986c306fa4ac3b9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.063627 20384 raft_consensus.cc:374] T 7a82ae1f4c66455caea352a304af2b51 P fceaf96557de46c3986c306fa4ac3b9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:33.063710 20384 raft_consensus.cc:729] T 7a82ae1f4c66455caea352a304af2b51 P fceaf96557de46c3986c306fa4ac3b9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fceaf96557de46c3986c306fa4ac3b9e, State: Initialized, Role: FOLLOWER
I20240503 14:13:33.063947 20384 consensus_queue.cc:260] T 7a82ae1f4c66455caea352a304af2b51 P fceaf96557de46c3986c306fa4ac3b9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.064631 20383 tablet_bootstrap.cc:492] T 220ef8bab4694ace9bdf0fce06226efa P 64ad8799c8df4b7589869baffb93b68d: No bootstrap required, opened a new log
I20240503 14:13:33.064652 20384 ts_tablet_manager.cc:1398] T 7a82ae1f4c66455caea352a304af2b51 P fceaf96557de46c3986c306fa4ac3b9e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:33.064806 20383 ts_tablet_manager.cc:1367] T 220ef8bab4694ace9bdf0fce06226efa P 64ad8799c8df4b7589869baffb93b68d: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.004s
I20240503 14:13:33.064962 20384 tablet_bootstrap.cc:492] T 220ef8bab4694ace9bdf0fce06226efa P fceaf96557de46c3986c306fa4ac3b9e: Bootstrap starting.
I20240503 14:13:33.065445 20383 raft_consensus.cc:348] T 220ef8bab4694ace9bdf0fce06226efa P 64ad8799c8df4b7589869baffb93b68d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.065622 20383 raft_consensus.cc:374] T 220ef8bab4694ace9bdf0fce06226efa P 64ad8799c8df4b7589869baffb93b68d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:33.065765 20383 raft_consensus.cc:729] T 220ef8bab4694ace9bdf0fce06226efa P 64ad8799c8df4b7589869baffb93b68d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64ad8799c8df4b7589869baffb93b68d, State: Initialized, Role: FOLLOWER
I20240503 14:13:33.066044 20383 consensus_queue.cc:260] T 220ef8bab4694ace9bdf0fce06226efa P 64ad8799c8df4b7589869baffb93b68d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.066347 20384 tablet_bootstrap.cc:654] T 220ef8bab4694ace9bdf0fce06226efa P fceaf96557de46c3986c306fa4ac3b9e: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:33.067437 20380 tablet_bootstrap.cc:492] T 220ef8bab4694ace9bdf0fce06226efa P a5d66df07715448fab64928f5bc8299a: No bootstrap required, opened a new log
I20240503 14:13:33.067592 20380 ts_tablet_manager.cc:1367] T 220ef8bab4694ace9bdf0fce06226efa P a5d66df07715448fab64928f5bc8299a: Time spent bootstrapping tablet: real 0.053s user 0.001s sys 0.003s
I20240503 14:13:33.068397 20380 raft_consensus.cc:348] T 220ef8bab4694ace9bdf0fce06226efa P a5d66df07715448fab64928f5bc8299a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.068620 20380 raft_consensus.cc:374] T 220ef8bab4694ace9bdf0fce06226efa P a5d66df07715448fab64928f5bc8299a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:33.068688 20380 raft_consensus.cc:729] T 220ef8bab4694ace9bdf0fce06226efa P a5d66df07715448fab64928f5bc8299a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5d66df07715448fab64928f5bc8299a, State: Initialized, Role: FOLLOWER
I20240503 14:13:33.068950 20380 consensus_queue.cc:260] T 220ef8bab4694ace9bdf0fce06226efa P a5d66df07715448fab64928f5bc8299a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.069545 20380 ts_tablet_manager.cc:1398] T 220ef8bab4694ace9bdf0fce06226efa P a5d66df07715448fab64928f5bc8299a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:13:33.069898 20380 tablet_bootstrap.cc:492] T 904450ab5e8b4250b7af18b98d2b0985 P a5d66df07715448fab64928f5bc8299a: Bootstrap starting.
I20240503 14:13:33.071671 20380 tablet_bootstrap.cc:654] T 904450ab5e8b4250b7af18b98d2b0985 P a5d66df07715448fab64928f5bc8299a: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:33.073213 20383 ts_tablet_manager.cc:1398] T 220ef8bab4694ace9bdf0fce06226efa P 64ad8799c8df4b7589869baffb93b68d: Time spent starting tablet: real 0.008s user 0.004s sys 0.003s
I20240503 14:13:33.073575 20384 tablet_bootstrap.cc:492] T 220ef8bab4694ace9bdf0fce06226efa P fceaf96557de46c3986c306fa4ac3b9e: No bootstrap required, opened a new log
I20240503 14:13:33.073733 20384 ts_tablet_manager.cc:1367] T 220ef8bab4694ace9bdf0fce06226efa P fceaf96557de46c3986c306fa4ac3b9e: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.003s
I20240503 14:13:33.074056 20383 tablet_bootstrap.cc:492] T 29f4ee03917646f6a5374217bed7a1be P 64ad8799c8df4b7589869baffb93b68d: Bootstrap starting.
I20240503 14:13:33.074368 20384 raft_consensus.cc:348] T 220ef8bab4694ace9bdf0fce06226efa P fceaf96557de46c3986c306fa4ac3b9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.074566 20384 raft_consensus.cc:374] T 220ef8bab4694ace9bdf0fce06226efa P fceaf96557de46c3986c306fa4ac3b9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:33.074627 20384 raft_consensus.cc:729] T 220ef8bab4694ace9bdf0fce06226efa P fceaf96557de46c3986c306fa4ac3b9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fceaf96557de46c3986c306fa4ac3b9e, State: Initialized, Role: FOLLOWER
I20240503 14:13:33.074851 20384 consensus_queue.cc:260] T 220ef8bab4694ace9bdf0fce06226efa P fceaf96557de46c3986c306fa4ac3b9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.075327 20384 ts_tablet_manager.cc:1398] T 220ef8bab4694ace9bdf0fce06226efa P fceaf96557de46c3986c306fa4ac3b9e: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:13:33.075438 20383 tablet_bootstrap.cc:654] T 29f4ee03917646f6a5374217bed7a1be P 64ad8799c8df4b7589869baffb93b68d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:33.075567 20384 tablet_bootstrap.cc:492] T ad711d78ee524d979867b65906abe456 P fceaf96557de46c3986c306fa4ac3b9e: Bootstrap starting.
I20240503 14:13:33.077103 20384 tablet_bootstrap.cc:654] T ad711d78ee524d979867b65906abe456 P fceaf96557de46c3986c306fa4ac3b9e: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:33.078920 20380 tablet_bootstrap.cc:492] T 904450ab5e8b4250b7af18b98d2b0985 P a5d66df07715448fab64928f5bc8299a: No bootstrap required, opened a new log
I20240503 14:13:33.079073 20380 ts_tablet_manager.cc:1367] T 904450ab5e8b4250b7af18b98d2b0985 P a5d66df07715448fab64928f5bc8299a: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.001s
I20240503 14:13:33.079890 20380 raft_consensus.cc:348] T 904450ab5e8b4250b7af18b98d2b0985 P a5d66df07715448fab64928f5bc8299a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.080106 20380 raft_consensus.cc:374] T 904450ab5e8b4250b7af18b98d2b0985 P a5d66df07715448fab64928f5bc8299a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:33.080170 20380 raft_consensus.cc:729] T 904450ab5e8b4250b7af18b98d2b0985 P a5d66df07715448fab64928f5bc8299a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5d66df07715448fab64928f5bc8299a, State: Initialized, Role: FOLLOWER
I20240503 14:13:33.080381 20380 consensus_queue.cc:260] T 904450ab5e8b4250b7af18b98d2b0985 P a5d66df07715448fab64928f5bc8299a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.080796 20380 ts_tablet_manager.cc:1398] T 904450ab5e8b4250b7af18b98d2b0985 P a5d66df07715448fab64928f5bc8299a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:13:33.080884 20383 tablet_bootstrap.cc:492] T 29f4ee03917646f6a5374217bed7a1be P 64ad8799c8df4b7589869baffb93b68d: No bootstrap required, opened a new log
I20240503 14:13:33.081017 20383 ts_tablet_manager.cc:1367] T 29f4ee03917646f6a5374217bed7a1be P 64ad8799c8df4b7589869baffb93b68d: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.001s
I20240503 14:13:33.081260 20380 tablet_bootstrap.cc:492] T 5484fffa7f6b4e64ba4798e8816b6c61 P a5d66df07715448fab64928f5bc8299a: Bootstrap starting.
I20240503 14:13:33.081308 20384 tablet_bootstrap.cc:492] T ad711d78ee524d979867b65906abe456 P fceaf96557de46c3986c306fa4ac3b9e: No bootstrap required, opened a new log
I20240503 14:13:33.081425 20384 ts_tablet_manager.cc:1367] T ad711d78ee524d979867b65906abe456 P fceaf96557de46c3986c306fa4ac3b9e: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240503 14:13:33.081679 20383 raft_consensus.cc:348] T 29f4ee03917646f6a5374217bed7a1be P 64ad8799c8df4b7589869baffb93b68d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.081909 20383 raft_consensus.cc:374] T 29f4ee03917646f6a5374217bed7a1be P 64ad8799c8df4b7589869baffb93b68d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:33.081990 20383 raft_consensus.cc:729] T 29f4ee03917646f6a5374217bed7a1be P 64ad8799c8df4b7589869baffb93b68d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64ad8799c8df4b7589869baffb93b68d, State: Initialized, Role: FOLLOWER
I20240503 14:13:33.082016 20384 raft_consensus.cc:348] T ad711d78ee524d979867b65906abe456 P fceaf96557de46c3986c306fa4ac3b9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.082217 20384 raft_consensus.cc:374] T ad711d78ee524d979867b65906abe456 P fceaf96557de46c3986c306fa4ac3b9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:33.082289 20384 raft_consensus.cc:729] T ad711d78ee524d979867b65906abe456 P fceaf96557de46c3986c306fa4ac3b9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fceaf96557de46c3986c306fa4ac3b9e, State: Initialized, Role: FOLLOWER
I20240503 14:13:33.082258 20383 consensus_queue.cc:260] T 29f4ee03917646f6a5374217bed7a1be P 64ad8799c8df4b7589869baffb93b68d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.082480 20384 consensus_queue.cc:260] T ad711d78ee524d979867b65906abe456 P fceaf96557de46c3986c306fa4ac3b9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.082660 20380 tablet_bootstrap.cc:654] T 5484fffa7f6b4e64ba4798e8816b6c61 P a5d66df07715448fab64928f5bc8299a: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:33.082767 20383 ts_tablet_manager.cc:1398] T 29f4ee03917646f6a5374217bed7a1be P 64ad8799c8df4b7589869baffb93b68d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:33.083048 20383 tablet_bootstrap.cc:492] T 44a4d2b820fc4d0da06c659fe98fe57c P 64ad8799c8df4b7589869baffb93b68d: Bootstrap starting.
I20240503 14:13:33.083632 20384 ts_tablet_manager.cc:1398] T ad711d78ee524d979867b65906abe456 P fceaf96557de46c3986c306fa4ac3b9e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:13:33.084002 20384 tablet_bootstrap.cc:492] T c45ceab3a36d4225aed74654ed530c9d P fceaf96557de46c3986c306fa4ac3b9e: Bootstrap starting.
I20240503 14:13:33.084342 20383 tablet_bootstrap.cc:654] T 44a4d2b820fc4d0da06c659fe98fe57c P 64ad8799c8df4b7589869baffb93b68d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:33.085826 20384 tablet_bootstrap.cc:654] T c45ceab3a36d4225aed74654ed530c9d P fceaf96557de46c3986c306fa4ac3b9e: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:33.090826 20380 tablet_bootstrap.cc:492] T 5484fffa7f6b4e64ba4798e8816b6c61 P a5d66df07715448fab64928f5bc8299a: No bootstrap required, opened a new log
I20240503 14:13:33.090987 20380 ts_tablet_manager.cc:1367] T 5484fffa7f6b4e64ba4798e8816b6c61 P a5d66df07715448fab64928f5bc8299a: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20240503 14:13:33.091804 20380 raft_consensus.cc:348] T 5484fffa7f6b4e64ba4798e8816b6c61 P a5d66df07715448fab64928f5bc8299a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.092097 20380 raft_consensus.cc:374] T 5484fffa7f6b4e64ba4798e8816b6c61 P a5d66df07715448fab64928f5bc8299a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:33.092218 20380 raft_consensus.cc:729] T 5484fffa7f6b4e64ba4798e8816b6c61 P a5d66df07715448fab64928f5bc8299a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5d66df07715448fab64928f5bc8299a, State: Initialized, Role: FOLLOWER
I20240503 14:13:33.092278 20383 tablet_bootstrap.cc:492] T 44a4d2b820fc4d0da06c659fe98fe57c P 64ad8799c8df4b7589869baffb93b68d: No bootstrap required, opened a new log
I20240503 14:13:33.092447 20383 ts_tablet_manager.cc:1367] T 44a4d2b820fc4d0da06c659fe98fe57c P 64ad8799c8df4b7589869baffb93b68d: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.003s
I20240503 14:13:33.092572 20380 consensus_queue.cc:260] T 5484fffa7f6b4e64ba4798e8816b6c61 P a5d66df07715448fab64928f5bc8299a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.092958 20383 raft_consensus.cc:348] T 44a4d2b820fc4d0da06c659fe98fe57c P 64ad8799c8df4b7589869baffb93b68d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.093176 20383 raft_consensus.cc:374] T 44a4d2b820fc4d0da06c659fe98fe57c P 64ad8799c8df4b7589869baffb93b68d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:33.093254 20383 raft_consensus.cc:729] T 44a4d2b820fc4d0da06c659fe98fe57c P 64ad8799c8df4b7589869baffb93b68d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64ad8799c8df4b7589869baffb93b68d, State: Initialized, Role: FOLLOWER
I20240503 14:13:33.093289 20380 ts_tablet_manager.cc:1398] T 5484fffa7f6b4e64ba4798e8816b6c61 P a5d66df07715448fab64928f5bc8299a: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240503 14:13:33.093573 20380 tablet_bootstrap.cc:492] T 3054a11a8da4470ea5241b92d164d6a8 P a5d66df07715448fab64928f5bc8299a: Bootstrap starting.
I20240503 14:13:33.093503 20383 consensus_queue.cc:260] T 44a4d2b820fc4d0da06c659fe98fe57c P 64ad8799c8df4b7589869baffb93b68d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.093780 20384 tablet_bootstrap.cc:492] T c45ceab3a36d4225aed74654ed530c9d P fceaf96557de46c3986c306fa4ac3b9e: No bootstrap required, opened a new log
I20240503 14:13:33.093904 20384 ts_tablet_manager.cc:1367] T c45ceab3a36d4225aed74654ed530c9d P fceaf96557de46c3986c306fa4ac3b9e: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.007s
I20240503 14:13:33.093973 20383 ts_tablet_manager.cc:1398] T 44a4d2b820fc4d0da06c659fe98fe57c P 64ad8799c8df4b7589869baffb93b68d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 14:13:33.094250 20383 tablet_bootstrap.cc:492] T 0eb7552695304a1ea6f8e02f189f7295 P 64ad8799c8df4b7589869baffb93b68d: Bootstrap starting.
I20240503 14:13:33.094467 20384 raft_consensus.cc:348] T c45ceab3a36d4225aed74654ed530c9d P fceaf96557de46c3986c306fa4ac3b9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.094635 20384 raft_consensus.cc:374] T c45ceab3a36d4225aed74654ed530c9d P fceaf96557de46c3986c306fa4ac3b9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:33.094705 20384 raft_consensus.cc:729] T c45ceab3a36d4225aed74654ed530c9d P fceaf96557de46c3986c306fa4ac3b9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fceaf96557de46c3986c306fa4ac3b9e, State: Initialized, Role: FOLLOWER
I20240503 14:13:33.095011 20384 consensus_queue.cc:260] T c45ceab3a36d4225aed74654ed530c9d P fceaf96557de46c3986c306fa4ac3b9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.095655 20383 tablet_bootstrap.cc:654] T 0eb7552695304a1ea6f8e02f189f7295 P 64ad8799c8df4b7589869baffb93b68d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:33.097334 20384 ts_tablet_manager.cc:1398] T c45ceab3a36d4225aed74654ed530c9d P fceaf96557de46c3986c306fa4ac3b9e: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 14:13:33.097379 20380 tablet_bootstrap.cc:654] T 3054a11a8da4470ea5241b92d164d6a8 P a5d66df07715448fab64928f5bc8299a: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:33.097672 20384 tablet_bootstrap.cc:492] T 904450ab5e8b4250b7af18b98d2b0985 P fceaf96557de46c3986c306fa4ac3b9e: Bootstrap starting.
I20240503 14:13:33.099351 20384 tablet_bootstrap.cc:654] T 904450ab5e8b4250b7af18b98d2b0985 P fceaf96557de46c3986c306fa4ac3b9e: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:33.104039 20383 tablet_bootstrap.cc:492] T 0eb7552695304a1ea6f8e02f189f7295 P 64ad8799c8df4b7589869baffb93b68d: No bootstrap required, opened a new log
I20240503 14:13:33.104167 20383 ts_tablet_manager.cc:1367] T 0eb7552695304a1ea6f8e02f189f7295 P 64ad8799c8df4b7589869baffb93b68d: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.006s
I20240503 14:13:33.104696 20383 raft_consensus.cc:348] T 0eb7552695304a1ea6f8e02f189f7295 P 64ad8799c8df4b7589869baffb93b68d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.104847 20383 raft_consensus.cc:374] T 0eb7552695304a1ea6f8e02f189f7295 P 64ad8799c8df4b7589869baffb93b68d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:33.104938 20383 raft_consensus.cc:729] T 0eb7552695304a1ea6f8e02f189f7295 P 64ad8799c8df4b7589869baffb93b68d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64ad8799c8df4b7589869baffb93b68d, State: Initialized, Role: FOLLOWER
I20240503 14:13:33.105144 20383 consensus_queue.cc:260] T 0eb7552695304a1ea6f8e02f189f7295 P 64ad8799c8df4b7589869baffb93b68d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.105543 20380 tablet_bootstrap.cc:492] T 3054a11a8da4470ea5241b92d164d6a8 P a5d66df07715448fab64928f5bc8299a: No bootstrap required, opened a new log
I20240503 14:13:33.105587 20383 ts_tablet_manager.cc:1398] T 0eb7552695304a1ea6f8e02f189f7295 P 64ad8799c8df4b7589869baffb93b68d: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240503 14:13:33.105808 20380 ts_tablet_manager.cc:1367] T 3054a11a8da4470ea5241b92d164d6a8 P a5d66df07715448fab64928f5bc8299a: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20240503 14:13:33.105959 20383 tablet_bootstrap.cc:492] T 5484fffa7f6b4e64ba4798e8816b6c61 P 64ad8799c8df4b7589869baffb93b68d: Bootstrap starting.
I20240503 14:13:33.106721 20380 raft_consensus.cc:348] T 3054a11a8da4470ea5241b92d164d6a8 P a5d66df07715448fab64928f5bc8299a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.107084 20380 raft_consensus.cc:374] T 3054a11a8da4470ea5241b92d164d6a8 P a5d66df07715448fab64928f5bc8299a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:33.107232 20380 raft_consensus.cc:729] T 3054a11a8da4470ea5241b92d164d6a8 P a5d66df07715448fab64928f5bc8299a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5d66df07715448fab64928f5bc8299a, State: Initialized, Role: FOLLOWER
I20240503 14:13:33.107466 20383 tablet_bootstrap.cc:654] T 5484fffa7f6b4e64ba4798e8816b6c61 P 64ad8799c8df4b7589869baffb93b68d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:33.107589 20380 consensus_queue.cc:260] T 3054a11a8da4470ea5241b92d164d6a8 P a5d66df07715448fab64928f5bc8299a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.109414 20380 ts_tablet_manager.cc:1398] T 3054a11a8da4470ea5241b92d164d6a8 P a5d66df07715448fab64928f5bc8299a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:33.109831 20380 tablet_bootstrap.cc:492] T 7905fd813a2d4a8aa8b2ac1e752b37dc P a5d66df07715448fab64928f5bc8299a: Bootstrap starting.
I20240503 14:13:33.110810 20382 raft_consensus.cc:482] T 44a4d2b820fc4d0da06c659fe98fe57c P a5d66df07715448fab64928f5bc8299a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:33.110989 20382 raft_consensus.cc:504] T 44a4d2b820fc4d0da06c659fe98fe57c P a5d66df07715448fab64928f5bc8299a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.111733 20380 tablet_bootstrap.cc:654] T 7905fd813a2d4a8aa8b2ac1e752b37dc P a5d66df07715448fab64928f5bc8299a: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:33.112210 20382 leader_election.cc:288] T 44a4d2b820fc4d0da06c659fe98fe57c P a5d66df07715448fab64928f5bc8299a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64ad8799c8df4b7589869baffb93b68d (127.16.94.129:37429), fceaf96557de46c3986c306fa4ac3b9e (127.16.94.131:39581)
I20240503 14:13:33.114109 20384 tablet_bootstrap.cc:492] T 904450ab5e8b4250b7af18b98d2b0985 P fceaf96557de46c3986c306fa4ac3b9e: No bootstrap required, opened a new log
I20240503 14:13:33.114269 20384 ts_tablet_manager.cc:1367] T 904450ab5e8b4250b7af18b98d2b0985 P fceaf96557de46c3986c306fa4ac3b9e: Time spent bootstrapping tablet: real 0.017s user 0.012s sys 0.003s
I20240503 14:13:33.115165 20384 raft_consensus.cc:348] T 904450ab5e8b4250b7af18b98d2b0985 P fceaf96557de46c3986c306fa4ac3b9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.115418 20384 raft_consensus.cc:374] T 904450ab5e8b4250b7af18b98d2b0985 P fceaf96557de46c3986c306fa4ac3b9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:33.115483 20384 raft_consensus.cc:729] T 904450ab5e8b4250b7af18b98d2b0985 P fceaf96557de46c3986c306fa4ac3b9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fceaf96557de46c3986c306fa4ac3b9e, State: Initialized, Role: FOLLOWER
I20240503 14:13:33.116446 20384 consensus_queue.cc:260] T 904450ab5e8b4250b7af18b98d2b0985 P fceaf96557de46c3986c306fa4ac3b9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.118187 20389 raft_consensus.cc:482] T 220ef8bab4694ace9bdf0fce06226efa P 64ad8799c8df4b7589869baffb93b68d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:33.118299 20389 raft_consensus.cc:504] T 220ef8bab4694ace9bdf0fce06226efa P 64ad8799c8df4b7589869baffb93b68d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.119277 20389 leader_election.cc:288] T 220ef8bab4694ace9bdf0fce06226efa P 64ad8799c8df4b7589869baffb93b68d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5d66df07715448fab64928f5bc8299a (127.16.94.130:39295), fceaf96557de46c3986c306fa4ac3b9e (127.16.94.131:39581)
I20240503 14:13:33.119642 20384 ts_tablet_manager.cc:1398] T 904450ab5e8b4250b7af18b98d2b0985 P fceaf96557de46c3986c306fa4ac3b9e: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 14:13:33.121745 20384 tablet_bootstrap.cc:492] T 5484fffa7f6b4e64ba4798e8816b6c61 P fceaf96557de46c3986c306fa4ac3b9e: Bootstrap starting.
I20240503 14:13:33.128171 20384 tablet_bootstrap.cc:654] T 5484fffa7f6b4e64ba4798e8816b6c61 P fceaf96557de46c3986c306fa4ac3b9e: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:33.132323 20335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44a4d2b820fc4d0da06c659fe98fe57c" candidate_uuid: "a5d66df07715448fab64928f5bc8299a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fceaf96557de46c3986c306fa4ac3b9e" is_pre_election: true
W20240503 14:13:33.133316 20226 leader_election.cc:341] T 44a4d2b820fc4d0da06c659fe98fe57c P a5d66df07715448fab64928f5bc8299a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fceaf96557de46c3986c306fa4ac3b9e (127.16.94.131:39581): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:33.139160 20185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44a4d2b820fc4d0da06c659fe98fe57c" candidate_uuid: "a5d66df07715448fab64928f5bc8299a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64ad8799c8df4b7589869baffb93b68d" is_pre_election: true
I20240503 14:13:33.139493 20185 raft_consensus.cc:2454] T 44a4d2b820fc4d0da06c659fe98fe57c P 64ad8799c8df4b7589869baffb93b68d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5d66df07715448fab64928f5bc8299a in term 0.
I20240503 14:13:33.140205 20224 leader_election.cc:302] T 44a4d2b820fc4d0da06c659fe98fe57c P a5d66df07715448fab64928f5bc8299a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 64ad8799c8df4b7589869baffb93b68d, a5d66df07715448fab64928f5bc8299a; no voters: fceaf96557de46c3986c306fa4ac3b9e
I20240503 14:13:33.140758 20382 raft_consensus.cc:2781] T 44a4d2b820fc4d0da06c659fe98fe57c P a5d66df07715448fab64928f5bc8299a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:33.140846 20382 raft_consensus.cc:482] T 44a4d2b820fc4d0da06c659fe98fe57c P a5d66df07715448fab64928f5bc8299a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:33.140925 20382 raft_consensus.cc:3037] T 44a4d2b820fc4d0da06c659fe98fe57c P a5d66df07715448fab64928f5bc8299a [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:33.144923 20382 raft_consensus.cc:504] T 44a4d2b820fc4d0da06c659fe98fe57c P a5d66df07715448fab64928f5bc8299a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.146685 20185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44a4d2b820fc4d0da06c659fe98fe57c" candidate_uuid: "a5d66df07715448fab64928f5bc8299a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64ad8799c8df4b7589869baffb93b68d"
I20240503 14:13:33.146906 20185 raft_consensus.cc:3037] T 44a4d2b820fc4d0da06c659fe98fe57c P 64ad8799c8df4b7589869baffb93b68d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:33.151219 20185 raft_consensus.cc:2454] T 44a4d2b820fc4d0da06c659fe98fe57c P 64ad8799c8df4b7589869baffb93b68d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5d66df07715448fab64928f5bc8299a in term 1.
I20240503 14:13:33.152230 20335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44a4d2b820fc4d0da06c659fe98fe57c" candidate_uuid: "a5d66df07715448fab64928f5bc8299a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fceaf96557de46c3986c306fa4ac3b9e"
W20240503 14:13:33.153090 20226 leader_election.cc:341] T 44a4d2b820fc4d0da06c659fe98fe57c P a5d66df07715448fab64928f5bc8299a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fceaf96557de46c3986c306fa4ac3b9e (127.16.94.131:39581): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:33.152408 20382 leader_election.cc:288] T 44a4d2b820fc4d0da06c659fe98fe57c P a5d66df07715448fab64928f5bc8299a [CANDIDATE]: Term 1 election: Requested vote from peers 64ad8799c8df4b7589869baffb93b68d (127.16.94.129:37429), fceaf96557de46c3986c306fa4ac3b9e (127.16.94.131:39581)
I20240503 14:13:33.153926 20224 leader_election.cc:302] T 44a4d2b820fc4d0da06c659fe98fe57c P a5d66df07715448fab64928f5bc8299a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 64ad8799c8df4b7589869baffb93b68d, a5d66df07715448fab64928f5bc8299a; no voters: fceaf96557de46c3986c306fa4ac3b9e
I20240503 14:13:33.154527 20382 raft_consensus.cc:2781] T 44a4d2b820fc4d0da06c659fe98fe57c P a5d66df07715448fab64928f5bc8299a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:33.156184 20380 tablet_bootstrap.cc:492] T 7905fd813a2d4a8aa8b2ac1e752b37dc P a5d66df07715448fab64928f5bc8299a: No bootstrap required, opened a new log
I20240503 14:13:33.156447 20380 ts_tablet_manager.cc:1367] T 7905fd813a2d4a8aa8b2ac1e752b37dc P a5d66df07715448fab64928f5bc8299a: Time spent bootstrapping tablet: real 0.047s user 0.003s sys 0.006s
I20240503 14:13:33.157480 20380 raft_consensus.cc:348] T 7905fd813a2d4a8aa8b2ac1e752b37dc P a5d66df07715448fab64928f5bc8299a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.158594 20380 raft_consensus.cc:374] T 7905fd813a2d4a8aa8b2ac1e752b37dc P a5d66df07715448fab64928f5bc8299a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:33.158825 20380 raft_consensus.cc:729] T 7905fd813a2d4a8aa8b2ac1e752b37dc P a5d66df07715448fab64928f5bc8299a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5d66df07715448fab64928f5bc8299a, State: Initialized, Role: FOLLOWER
I20240503 14:13:33.159278 20380 consensus_queue.cc:260] T 7905fd813a2d4a8aa8b2ac1e752b37dc P a5d66df07715448fab64928f5bc8299a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.163661 20382 raft_consensus.cc:686] T 44a4d2b820fc4d0da06c659fe98fe57c P a5d66df07715448fab64928f5bc8299a [term 1 LEADER]: Becoming Leader. State: Replica: a5d66df07715448fab64928f5bc8299a, State: Running, Role: LEADER
I20240503 14:13:33.164013 20382 consensus_queue.cc:237] T 44a4d2b820fc4d0da06c659fe98fe57c P a5d66df07715448fab64928f5bc8299a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.164443 20383 tablet_bootstrap.cc:492] T 5484fffa7f6b4e64ba4798e8816b6c61 P 64ad8799c8df4b7589869baffb93b68d: No bootstrap required, opened a new log
I20240503 14:13:33.164646 20383 ts_tablet_manager.cc:1367] T 5484fffa7f6b4e64ba4798e8816b6c61 P 64ad8799c8df4b7589869baffb93b68d: Time spent bootstrapping tablet: real 0.059s user 0.012s sys 0.045s
I20240503 14:13:33.165515 20383 raft_consensus.cc:348] T 5484fffa7f6b4e64ba4798e8816b6c61 P 64ad8799c8df4b7589869baffb93b68d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.165889 20383 raft_consensus.cc:374] T 5484fffa7f6b4e64ba4798e8816b6c61 P 64ad8799c8df4b7589869baffb93b68d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:33.166023 20383 raft_consensus.cc:729] T 5484fffa7f6b4e64ba4798e8816b6c61 P 64ad8799c8df4b7589869baffb93b68d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64ad8799c8df4b7589869baffb93b68d, State: Initialized, Role: FOLLOWER
I20240503 14:13:33.166323 20383 consensus_queue.cc:260] T 5484fffa7f6b4e64ba4798e8816b6c61 P 64ad8799c8df4b7589869baffb93b68d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.167022 20383 ts_tablet_manager.cc:1398] T 5484fffa7f6b4e64ba4798e8816b6c61 P 64ad8799c8df4b7589869baffb93b68d: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 14:13:33.167363 20384 tablet_bootstrap.cc:492] T 5484fffa7f6b4e64ba4798e8816b6c61 P fceaf96557de46c3986c306fa4ac3b9e: No bootstrap required, opened a new log
I20240503 14:13:33.167590 20384 ts_tablet_manager.cc:1367] T 5484fffa7f6b4e64ba4798e8816b6c61 P fceaf96557de46c3986c306fa4ac3b9e: Time spent bootstrapping tablet: real 0.046s user 0.009s sys 0.030s
I20240503 14:13:33.168224 20383 tablet_bootstrap.cc:492] T ad711d78ee524d979867b65906abe456 P 64ad8799c8df4b7589869baffb93b68d: Bootstrap starting.
I20240503 14:13:33.168397 20384 raft_consensus.cc:348] T 5484fffa7f6b4e64ba4798e8816b6c61 P fceaf96557de46c3986c306fa4ac3b9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.168673 20384 raft_consensus.cc:374] T 5484fffa7f6b4e64ba4798e8816b6c61 P fceaf96557de46c3986c306fa4ac3b9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:33.168813 20384 raft_consensus.cc:729] T 5484fffa7f6b4e64ba4798e8816b6c61 P fceaf96557de46c3986c306fa4ac3b9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fceaf96557de46c3986c306fa4ac3b9e, State: Initialized, Role: FOLLOWER
I20240503 14:13:33.169081 20384 consensus_queue.cc:260] T 5484fffa7f6b4e64ba4798e8816b6c61 P fceaf96557de46c3986c306fa4ac3b9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.169837 20384 ts_tablet_manager.cc:1398] T 5484fffa7f6b4e64ba4798e8816b6c61 P fceaf96557de46c3986c306fa4ac3b9e: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:13:33.170185 20384 tablet_bootstrap.cc:492] T 3054a11a8da4470ea5241b92d164d6a8 P fceaf96557de46c3986c306fa4ac3b9e: Bootstrap starting.
I20240503 14:13:33.170365 20383 tablet_bootstrap.cc:654] T ad711d78ee524d979867b65906abe456 P 64ad8799c8df4b7589869baffb93b68d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:33.171509 20384 tablet_bootstrap.cc:654] T 3054a11a8da4470ea5241b92d164d6a8 P fceaf96557de46c3986c306fa4ac3b9e: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:33.175093 20380 ts_tablet_manager.cc:1398] T 7905fd813a2d4a8aa8b2ac1e752b37dc P a5d66df07715448fab64928f5bc8299a: Time spent starting tablet: real 0.018s user 0.005s sys 0.003s
I20240503 14:13:33.175494 20380 tablet_bootstrap.cc:492] T c45ceab3a36d4225aed74654ed530c9d P a5d66df07715448fab64928f5bc8299a: Bootstrap starting.
I20240503 14:13:33.176612 20388 raft_consensus.cc:482] T ad711d78ee524d979867b65906abe456 P fceaf96557de46c3986c306fa4ac3b9e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:33.176787 20388 raft_consensus.cc:504] T ad711d78ee524d979867b65906abe456 P fceaf96557de46c3986c306fa4ac3b9e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.177840 20388 leader_election.cc:288] T ad711d78ee524d979867b65906abe456 P fceaf96557de46c3986c306fa4ac3b9e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64ad8799c8df4b7589869baffb93b68d (127.16.94.129:37429), a5d66df07715448fab64928f5bc8299a (127.16.94.130:39295)
I20240503 14:13:33.179047 20380 tablet_bootstrap.cc:654] T c45ceab3a36d4225aed74654ed530c9d P a5d66df07715448fab64928f5bc8299a: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:33.180058 20083 catalog_manager.cc:5484] T 44a4d2b820fc4d0da06c659fe98fe57c P a5d66df07715448fab64928f5bc8299a reported cstate change: term changed from 0 to 1, leader changed from <none> to a5d66df07715448fab64928f5bc8299a (127.16.94.130). New cstate: current_term: 1 leader_uuid: "a5d66df07715448fab64928f5bc8299a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:33.182739 20384 tablet_bootstrap.cc:492] T 3054a11a8da4470ea5241b92d164d6a8 P fceaf96557de46c3986c306fa4ac3b9e: No bootstrap required, opened a new log
I20240503 14:13:33.182886 20384 ts_tablet_manager.cc:1367] T 3054a11a8da4470ea5241b92d164d6a8 P fceaf96557de46c3986c306fa4ac3b9e: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.003s
I20240503 14:13:33.183579 20384 raft_consensus.cc:348] T 3054a11a8da4470ea5241b92d164d6a8 P fceaf96557de46c3986c306fa4ac3b9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.183795 20384 raft_consensus.cc:374] T 3054a11a8da4470ea5241b92d164d6a8 P fceaf96557de46c3986c306fa4ac3b9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:33.183868 20384 raft_consensus.cc:729] T 3054a11a8da4470ea5241b92d164d6a8 P fceaf96557de46c3986c306fa4ac3b9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fceaf96557de46c3986c306fa4ac3b9e, State: Initialized, Role: FOLLOWER
I20240503 14:13:33.184082 20384 consensus_queue.cc:260] T 3054a11a8da4470ea5241b92d164d6a8 P fceaf96557de46c3986c306fa4ac3b9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.187439 20380 tablet_bootstrap.cc:492] T c45ceab3a36d4225aed74654ed530c9d P a5d66df07715448fab64928f5bc8299a: No bootstrap required, opened a new log
I20240503 14:13:33.187723 20380 ts_tablet_manager.cc:1367] T c45ceab3a36d4225aed74654ed530c9d P a5d66df07715448fab64928f5bc8299a: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20240503 14:13:33.188570 20384 ts_tablet_manager.cc:1398] T 3054a11a8da4470ea5241b92d164d6a8 P fceaf96557de46c3986c306fa4ac3b9e: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240503 14:13:33.190102 20384 tablet_bootstrap.cc:492] T 0eb7552695304a1ea6f8e02f189f7295 P fceaf96557de46c3986c306fa4ac3b9e: Bootstrap starting.
I20240503 14:13:33.192255 20384 tablet_bootstrap.cc:654] T 0eb7552695304a1ea6f8e02f189f7295 P fceaf96557de46c3986c306fa4ac3b9e: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:33.188724 20380 raft_consensus.cc:348] T c45ceab3a36d4225aed74654ed530c9d P a5d66df07715448fab64928f5bc8299a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.193543 20380 raft_consensus.cc:374] T c45ceab3a36d4225aed74654ed530c9d P a5d66df07715448fab64928f5bc8299a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:33.193701 20380 raft_consensus.cc:729] T c45ceab3a36d4225aed74654ed530c9d P a5d66df07715448fab64928f5bc8299a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5d66df07715448fab64928f5bc8299a, State: Initialized, Role: FOLLOWER
I20240503 14:13:33.193994 20380 consensus_queue.cc:260] T c45ceab3a36d4225aed74654ed530c9d P a5d66df07715448fab64928f5bc8299a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.194828 20380 ts_tablet_manager.cc:1398] T c45ceab3a36d4225aed74654ed530c9d P a5d66df07715448fab64928f5bc8299a: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 14:13:33.195230 20380 tablet_bootstrap.cc:492] T 29f4ee03917646f6a5374217bed7a1be P a5d66df07715448fab64928f5bc8299a: Bootstrap starting.
I20240503 14:13:33.197335 20380 tablet_bootstrap.cc:654] T 29f4ee03917646f6a5374217bed7a1be P a5d66df07715448fab64928f5bc8299a: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:33.199117 20383 tablet_bootstrap.cc:492] T ad711d78ee524d979867b65906abe456 P 64ad8799c8df4b7589869baffb93b68d: No bootstrap required, opened a new log
I20240503 14:13:33.199369 20383 ts_tablet_manager.cc:1367] T ad711d78ee524d979867b65906abe456 P 64ad8799c8df4b7589869baffb93b68d: Time spent bootstrapping tablet: real 0.031s user 0.007s sys 0.023s
I20240503 14:13:33.200177 20261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "220ef8bab4694ace9bdf0fce06226efa" candidate_uuid: "64ad8799c8df4b7589869baffb93b68d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5d66df07715448fab64928f5bc8299a" is_pre_election: true
I20240503 14:13:33.200522 20261 raft_consensus.cc:2454] T 220ef8bab4694ace9bdf0fce06226efa P a5d66df07715448fab64928f5bc8299a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64ad8799c8df4b7589869baffb93b68d in term 0.
I20240503 14:13:33.200973 20384 tablet_bootstrap.cc:492] T 0eb7552695304a1ea6f8e02f189f7295 P fceaf96557de46c3986c306fa4ac3b9e: No bootstrap required, opened a new log
I20240503 14:13:33.201133 20384 ts_tablet_manager.cc:1367] T 0eb7552695304a1ea6f8e02f189f7295 P fceaf96557de46c3986c306fa4ac3b9e: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.002s
I20240503 14:13:33.201321 20151 leader_election.cc:302] T 220ef8bab4694ace9bdf0fce06226efa P 64ad8799c8df4b7589869baffb93b68d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64ad8799c8df4b7589869baffb93b68d, a5d66df07715448fab64928f5bc8299a; no voters:
I20240503 14:13:33.201745 20389 raft_consensus.cc:2781] T 220ef8bab4694ace9bdf0fce06226efa P 64ad8799c8df4b7589869baffb93b68d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:33.201851 20389 raft_consensus.cc:482] T 220ef8bab4694ace9bdf0fce06226efa P 64ad8799c8df4b7589869baffb93b68d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:33.201946 20389 raft_consensus.cc:3037] T 220ef8bab4694ace9bdf0fce06226efa P 64ad8799c8df4b7589869baffb93b68d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:33.201990 20384 raft_consensus.cc:348] T 0eb7552695304a1ea6f8e02f189f7295 P fceaf96557de46c3986c306fa4ac3b9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.202359 20384 raft_consensus.cc:374] T 0eb7552695304a1ea6f8e02f189f7295 P fceaf96557de46c3986c306fa4ac3b9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:33.202559 20384 raft_consensus.cc:729] T 0eb7552695304a1ea6f8e02f189f7295 P fceaf96557de46c3986c306fa4ac3b9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fceaf96557de46c3986c306fa4ac3b9e, State: Initialized, Role: FOLLOWER
I20240503 14:13:33.202980 20384 consensus_queue.cc:260] T 0eb7552695304a1ea6f8e02f189f7295 P fceaf96557de46c3986c306fa4ac3b9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.204000 20384 ts_tablet_manager.cc:1398] T 0eb7552695304a1ea6f8e02f189f7295 P fceaf96557de46c3986c306fa4ac3b9e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 14:13:33.204422 20384 tablet_bootstrap.cc:492] T 29f4ee03917646f6a5374217bed7a1be P fceaf96557de46c3986c306fa4ac3b9e: Bootstrap starting.
I20240503 14:13:33.205328 20389 raft_consensus.cc:504] T 220ef8bab4694ace9bdf0fce06226efa P 64ad8799c8df4b7589869baffb93b68d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.206077 20389 leader_election.cc:288] T 220ef8bab4694ace9bdf0fce06226efa P 64ad8799c8df4b7589869baffb93b68d [CANDIDATE]: Term 1 election: Requested vote from peers a5d66df07715448fab64928f5bc8299a (127.16.94.130:39295), fceaf96557de46c3986c306fa4ac3b9e (127.16.94.131:39581)
I20240503 14:13:33.206418 20383 raft_consensus.cc:348] T ad711d78ee524d979867b65906abe456 P 64ad8799c8df4b7589869baffb93b68d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.206626 20383 raft_consensus.cc:374] T ad711d78ee524d979867b65906abe456 P 64ad8799c8df4b7589869baffb93b68d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:33.206560 20261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "220ef8bab4694ace9bdf0fce06226efa" candidate_uuid: "64ad8799c8df4b7589869baffb93b68d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5d66df07715448fab64928f5bc8299a"
I20240503 14:13:33.206708 20383 raft_consensus.cc:729] T ad711d78ee524d979867b65906abe456 P 64ad8799c8df4b7589869baffb93b68d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64ad8799c8df4b7589869baffb93b68d, State: Initialized, Role: FOLLOWER
I20240503 14:13:33.206830 20261 raft_consensus.cc:3037] T 220ef8bab4694ace9bdf0fce06226efa P a5d66df07715448fab64928f5bc8299a [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:33.206941 20383 consensus_queue.cc:260] T ad711d78ee524d979867b65906abe456 P 64ad8799c8df4b7589869baffb93b68d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.208179 20383 ts_tablet_manager.cc:1398] T ad711d78ee524d979867b65906abe456 P 64ad8799c8df4b7589869baffb93b68d: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20240503 14:13:33.208214 20185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad711d78ee524d979867b65906abe456" candidate_uuid: "fceaf96557de46c3986c306fa4ac3b9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64ad8799c8df4b7589869baffb93b68d" is_pre_election: true
I20240503 14:13:33.208385 20185 raft_consensus.cc:2454] T ad711d78ee524d979867b65906abe456 P 64ad8799c8df4b7589869baffb93b68d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fceaf96557de46c3986c306fa4ac3b9e in term 0.
I20240503 14:13:33.208459 20383 tablet_bootstrap.cc:492] T d4743d5a73c54086b277f4c4b1a248fd P 64ad8799c8df4b7589869baffb93b68d: Bootstrap starting.
I20240503 14:13:33.208817 20298 leader_election.cc:302] T ad711d78ee524d979867b65906abe456 P fceaf96557de46c3986c306fa4ac3b9e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64ad8799c8df4b7589869baffb93b68d, fceaf96557de46c3986c306fa4ac3b9e; no voters:
I20240503 14:13:33.209067 20388 raft_consensus.cc:2781] T ad711d78ee524d979867b65906abe456 P fceaf96557de46c3986c306fa4ac3b9e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:33.209194 20388 raft_consensus.cc:482] T ad711d78ee524d979867b65906abe456 P fceaf96557de46c3986c306fa4ac3b9e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:33.209295 20388 raft_consensus.cc:3037] T ad711d78ee524d979867b65906abe456 P fceaf96557de46c3986c306fa4ac3b9e [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:33.210245 20383 tablet_bootstrap.cc:654] T d4743d5a73c54086b277f4c4b1a248fd P 64ad8799c8df4b7589869baffb93b68d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:33.212234 20261 raft_consensus.cc:2454] T 220ef8bab4694ace9bdf0fce06226efa P a5d66df07715448fab64928f5bc8299a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64ad8799c8df4b7589869baffb93b68d in term 1.
I20240503 14:13:33.212786 20388 raft_consensus.cc:504] T ad711d78ee524d979867b65906abe456 P fceaf96557de46c3986c306fa4ac3b9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.217944 20388 leader_election.cc:288] T ad711d78ee524d979867b65906abe456 P fceaf96557de46c3986c306fa4ac3b9e [CANDIDATE]: Term 1 election: Requested vote from peers 64ad8799c8df4b7589869baffb93b68d (127.16.94.129:37429), a5d66df07715448fab64928f5bc8299a (127.16.94.130:39295)
I20240503 14:13:33.214560 20335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "220ef8bab4694ace9bdf0fce06226efa" candidate_uuid: "64ad8799c8df4b7589869baffb93b68d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fceaf96557de46c3986c306fa4ac3b9e" is_pre_election: true
I20240503 14:13:33.218042 20420 raft_consensus.cc:482] T 7a82ae1f4c66455caea352a304af2b51 P fceaf96557de46c3986c306fa4ac3b9e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:33.215003 20334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "220ef8bab4694ace9bdf0fce06226efa" candidate_uuid: "64ad8799c8df4b7589869baffb93b68d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fceaf96557de46c3986c306fa4ac3b9e"
I20240503 14:13:33.218209 20335 raft_consensus.cc:2454] T 220ef8bab4694ace9bdf0fce06226efa P fceaf96557de46c3986c306fa4ac3b9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64ad8799c8df4b7589869baffb93b68d in term 0.
I20240503 14:13:33.216133 20382 raft_consensus.cc:482] T 904450ab5e8b4250b7af18b98d2b0985 P a5d66df07715448fab64928f5bc8299a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:33.218466 20382 raft_consensus.cc:504] T 904450ab5e8b4250b7af18b98d2b0985 P a5d66df07715448fab64928f5bc8299a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.218967 20185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad711d78ee524d979867b65906abe456" candidate_uuid: "fceaf96557de46c3986c306fa4ac3b9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64ad8799c8df4b7589869baffb93b68d"
I20240503 14:13:33.219116 20382 leader_election.cc:288] T 904450ab5e8b4250b7af18b98d2b0985 P a5d66df07715448fab64928f5bc8299a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64ad8799c8df4b7589869baffb93b68d (127.16.94.129:37429), fceaf96557de46c3986c306fa4ac3b9e (127.16.94.131:39581)
I20240503 14:13:33.219200 20185 raft_consensus.cc:3037] T ad711d78ee524d979867b65906abe456 P 64ad8799c8df4b7589869baffb93b68d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:33.219825 20151 leader_election.cc:302] T 220ef8bab4694ace9bdf0fce06226efa P 64ad8799c8df4b7589869baffb93b68d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 64ad8799c8df4b7589869baffb93b68d, a5d66df07715448fab64928f5bc8299a; no voters: fceaf96557de46c3986c306fa4ac3b9e
I20240503 14:13:33.220454 20384 tablet_bootstrap.cc:654] T 29f4ee03917646f6a5374217bed7a1be P fceaf96557de46c3986c306fa4ac3b9e: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:33.220336 20184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904450ab5e8b4250b7af18b98d2b0985" candidate_uuid: "a5d66df07715448fab64928f5bc8299a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64ad8799c8df4b7589869baffb93b68d" is_pre_election: true
I20240503 14:13:33.220798 20389 raft_consensus.cc:2781] T 220ef8bab4694ace9bdf0fce06226efa P 64ad8799c8df4b7589869baffb93b68d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:33.216817 20380 tablet_bootstrap.cc:492] T 29f4ee03917646f6a5374217bed7a1be P a5d66df07715448fab64928f5bc8299a: No bootstrap required, opened a new log
W20240503 14:13:33.221794 20224 leader_election.cc:341] T 904450ab5e8b4250b7af18b98d2b0985 P a5d66df07715448fab64928f5bc8299a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 64ad8799c8df4b7589869baffb93b68d (127.16.94.129:37429): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:33.221803 20380 ts_tablet_manager.cc:1367] T 29f4ee03917646f6a5374217bed7a1be P a5d66df07715448fab64928f5bc8299a: Time spent bootstrapping tablet: real 0.027s user 0.013s sys 0.007s
I20240503 14:13:33.222831 20334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904450ab5e8b4250b7af18b98d2b0985" candidate_uuid: "a5d66df07715448fab64928f5bc8299a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fceaf96557de46c3986c306fa4ac3b9e" is_pre_election: true
I20240503 14:13:33.224277 20334 raft_consensus.cc:2454] T 904450ab5e8b4250b7af18b98d2b0985 P fceaf96557de46c3986c306fa4ac3b9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5d66df07715448fab64928f5bc8299a in term 0.
I20240503 14:13:33.224678 20226 leader_election.cc:302] T 904450ab5e8b4250b7af18b98d2b0985 P a5d66df07715448fab64928f5bc8299a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a5d66df07715448fab64928f5bc8299a, fceaf96557de46c3986c306fa4ac3b9e; no voters: 64ad8799c8df4b7589869baffb93b68d
I20240503 14:13:33.224910 20380 raft_consensus.cc:348] T 29f4ee03917646f6a5374217bed7a1be P a5d66df07715448fab64928f5bc8299a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.225126 20380 raft_consensus.cc:374] T 29f4ee03917646f6a5374217bed7a1be P a5d66df07715448fab64928f5bc8299a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:33.225157 20382 raft_consensus.cc:2781] T 904450ab5e8b4250b7af18b98d2b0985 P a5d66df07715448fab64928f5bc8299a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:33.225268 20380 raft_consensus.cc:729] T 29f4ee03917646f6a5374217bed7a1be P a5d66df07715448fab64928f5bc8299a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5d66df07715448fab64928f5bc8299a, State: Initialized, Role: FOLLOWER
I20240503 14:13:33.225334 20382 raft_consensus.cc:482] T 904450ab5e8b4250b7af18b98d2b0985 P a5d66df07715448fab64928f5bc8299a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:33.225430 20382 raft_consensus.cc:3037] T 904450ab5e8b4250b7af18b98d2b0985 P a5d66df07715448fab64928f5bc8299a [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:33.225590 20380 consensus_queue.cc:260] T 29f4ee03917646f6a5374217bed7a1be P a5d66df07715448fab64928f5bc8299a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.218205 20420 raft_consensus.cc:504] T 7a82ae1f4c66455caea352a304af2b51 P fceaf96557de46c3986c306fa4ac3b9e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.229501 20380 ts_tablet_manager.cc:1398] T 29f4ee03917646f6a5374217bed7a1be P a5d66df07715448fab64928f5bc8299a: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 14:13:33.230832 20380 tablet_bootstrap.cc:492] T d4743d5a73c54086b277f4c4b1a248fd P a5d66df07715448fab64928f5bc8299a: Bootstrap starting.
I20240503 14:13:33.231158 20185 raft_consensus.cc:2454] T ad711d78ee524d979867b65906abe456 P 64ad8799c8df4b7589869baffb93b68d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fceaf96557de46c3986c306fa4ac3b9e in term 1.
I20240503 14:13:33.232770 20298 leader_election.cc:302] T ad711d78ee524d979867b65906abe456 P fceaf96557de46c3986c306fa4ac3b9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64ad8799c8df4b7589869baffb93b68d, fceaf96557de46c3986c306fa4ac3b9e; no voters:
I20240503 14:13:33.232774 20389 raft_consensus.cc:686] T 220ef8bab4694ace9bdf0fce06226efa P 64ad8799c8df4b7589869baffb93b68d [term 1 LEADER]: Becoming Leader. State: Replica: 64ad8799c8df4b7589869baffb93b68d, State: Running, Role: LEADER
I20240503 14:13:33.233059 20420 leader_election.cc:288] T 7a82ae1f4c66455caea352a304af2b51 P fceaf96557de46c3986c306fa4ac3b9e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64ad8799c8df4b7589869baffb93b68d (127.16.94.129:37429), a5d66df07715448fab64928f5bc8299a (127.16.94.130:39295)
I20240503 14:13:33.233071 20388 raft_consensus.cc:2781] T ad711d78ee524d979867b65906abe456 P fceaf96557de46c3986c306fa4ac3b9e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:33.233410 20383 tablet_bootstrap.cc:492] T d4743d5a73c54086b277f4c4b1a248fd P 64ad8799c8df4b7589869baffb93b68d: No bootstrap required, opened a new log
I20240503 14:13:33.233453 20388 raft_consensus.cc:686] T ad711d78ee524d979867b65906abe456 P fceaf96557de46c3986c306fa4ac3b9e [term 1 LEADER]: Becoming Leader. State: Replica: fceaf96557de46c3986c306fa4ac3b9e, State: Running, Role: LEADER
I20240503 14:13:33.233731 20383 ts_tablet_manager.cc:1367] T d4743d5a73c54086b277f4c4b1a248fd P 64ad8799c8df4b7589869baffb93b68d: Time spent bootstrapping tablet: real 0.025s user 0.008s sys 0.004s
I20240503 14:13:33.233827 20388 consensus_queue.cc:237] T ad711d78ee524d979867b65906abe456 P fceaf96557de46c3986c306fa4ac3b9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.233868 20260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad711d78ee524d979867b65906abe456" candidate_uuid: "fceaf96557de46c3986c306fa4ac3b9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5d66df07715448fab64928f5bc8299a"
I20240503 14:13:33.234575 20389 consensus_queue.cc:237] T 220ef8bab4694ace9bdf0fce06226efa P 64ad8799c8df4b7589869baffb93b68d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.234915 20259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a82ae1f4c66455caea352a304af2b51" candidate_uuid: "fceaf96557de46c3986c306fa4ac3b9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5d66df07715448fab64928f5bc8299a" is_pre_election: true
I20240503 14:13:33.235934 20185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a82ae1f4c66455caea352a304af2b51" candidate_uuid: "fceaf96557de46c3986c306fa4ac3b9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64ad8799c8df4b7589869baffb93b68d" is_pre_election: true
I20240503 14:13:33.236081 20261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad711d78ee524d979867b65906abe456" candidate_uuid: "fceaf96557de46c3986c306fa4ac3b9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5d66df07715448fab64928f5bc8299a" is_pre_election: true
W20240503 14:13:33.236717 20301 leader_election.cc:341] T ad711d78ee524d979867b65906abe456 P fceaf96557de46c3986c306fa4ac3b9e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a5d66df07715448fab64928f5bc8299a (127.16.94.130:39295): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:13:33.237200 20298 leader_election.cc:341] T 7a82ae1f4c66455caea352a304af2b51 P fceaf96557de46c3986c306fa4ac3b9e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 64ad8799c8df4b7589869baffb93b68d (127.16.94.129:37429): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:13:33.241758 20301 leader_election.cc:341] T 7a82ae1f4c66455caea352a304af2b51 P fceaf96557de46c3986c306fa4ac3b9e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a5d66df07715448fab64928f5bc8299a (127.16.94.130:39295): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:33.241911 20301 leader_election.cc:302] T 7a82ae1f4c66455caea352a304af2b51 P fceaf96557de46c3986c306fa4ac3b9e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fceaf96557de46c3986c306fa4ac3b9e; no voters: 64ad8799c8df4b7589869baffb93b68d, a5d66df07715448fab64928f5bc8299a
W20240503 14:13:33.242321 20301 leader_election.cc:341] T ad711d78ee524d979867b65906abe456 P fceaf96557de46c3986c306fa4ac3b9e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a5d66df07715448fab64928f5bc8299a (127.16.94.130:39295): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:33.242399 20380 tablet_bootstrap.cc:654] T d4743d5a73c54086b277f4c4b1a248fd P a5d66df07715448fab64928f5bc8299a: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:33.242697 20420 raft_consensus.cc:2726] T 7a82ae1f4c66455caea352a304af2b51 P fceaf96557de46c3986c306fa4ac3b9e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:13:33.244755 20382 raft_consensus.cc:504] T 904450ab5e8b4250b7af18b98d2b0985 P a5d66df07715448fab64928f5bc8299a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.246861 20334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904450ab5e8b4250b7af18b98d2b0985" candidate_uuid: "a5d66df07715448fab64928f5bc8299a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fceaf96557de46c3986c306fa4ac3b9e"
I20240503 14:13:33.246943 20383 raft_consensus.cc:348] T d4743d5a73c54086b277f4c4b1a248fd P 64ad8799c8df4b7589869baffb93b68d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.247135 20334 raft_consensus.cc:3037] T 904450ab5e8b4250b7af18b98d2b0985 P fceaf96557de46c3986c306fa4ac3b9e [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:33.247166 20383 raft_consensus.cc:374] T d4743d5a73c54086b277f4c4b1a248fd P 64ad8799c8df4b7589869baffb93b68d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:33.247249 20383 raft_consensus.cc:729] T d4743d5a73c54086b277f4c4b1a248fd P 64ad8799c8df4b7589869baffb93b68d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64ad8799c8df4b7589869baffb93b68d, State: Initialized, Role: FOLLOWER
I20240503 14:13:33.250218 20384 tablet_bootstrap.cc:492] T 29f4ee03917646f6a5374217bed7a1be P fceaf96557de46c3986c306fa4ac3b9e: No bootstrap required, opened a new log
I20240503 14:13:33.250311 20382 leader_election.cc:288] T 904450ab5e8b4250b7af18b98d2b0985 P a5d66df07715448fab64928f5bc8299a [CANDIDATE]: Term 1 election: Requested vote from peers 64ad8799c8df4b7589869baffb93b68d (127.16.94.129:37429), fceaf96557de46c3986c306fa4ac3b9e (127.16.94.131:39581)
I20240503 14:13:33.249996 20383 consensus_queue.cc:260] T d4743d5a73c54086b277f4c4b1a248fd P 64ad8799c8df4b7589869baffb93b68d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.251230 20383 ts_tablet_manager.cc:1398] T d4743d5a73c54086b277f4c4b1a248fd P 64ad8799c8df4b7589869baffb93b68d: Time spent starting tablet: real 0.017s user 0.000s sys 0.002s
I20240503 14:13:33.251549 20185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904450ab5e8b4250b7af18b98d2b0985" candidate_uuid: "a5d66df07715448fab64928f5bc8299a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64ad8799c8df4b7589869baffb93b68d"
W20240503 14:13:33.252053 20224 leader_election.cc:341] T 904450ab5e8b4250b7af18b98d2b0985 P a5d66df07715448fab64928f5bc8299a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 64ad8799c8df4b7589869baffb93b68d (127.16.94.129:37429): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:33.252418 20384 ts_tablet_manager.cc:1367] T 29f4ee03917646f6a5374217bed7a1be P fceaf96557de46c3986c306fa4ac3b9e: Time spent bootstrapping tablet: real 0.046s user 0.002s sys 0.005s
I20240503 14:13:33.252065 20083 catalog_manager.cc:5484] T ad711d78ee524d979867b65906abe456 P fceaf96557de46c3986c306fa4ac3b9e reported cstate change: term changed from 0 to 1, leader changed from <none> to fceaf96557de46c3986c306fa4ac3b9e (127.16.94.131). New cstate: current_term: 1 leader_uuid: "fceaf96557de46c3986c306fa4ac3b9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } health_report { overall_health: HEALTHY } } }
I20240503 14:13:33.254595 20383 tablet_bootstrap.cc:492] T 3054a11a8da4470ea5241b92d164d6a8 P 64ad8799c8df4b7589869baffb93b68d: Bootstrap starting.
I20240503 14:13:33.253702 20082 catalog_manager.cc:5484] T 220ef8bab4694ace9bdf0fce06226efa P 64ad8799c8df4b7589869baffb93b68d reported cstate change: term changed from 0 to 1, leader changed from <none> to 64ad8799c8df4b7589869baffb93b68d (127.16.94.129). New cstate: current_term: 1 leader_uuid: "64ad8799c8df4b7589869baffb93b68d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:33.257531 20383 tablet_bootstrap.cc:654] T 3054a11a8da4470ea5241b92d164d6a8 P 64ad8799c8df4b7589869baffb93b68d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:33.257627 20384 raft_consensus.cc:348] T 29f4ee03917646f6a5374217bed7a1be P fceaf96557de46c3986c306fa4ac3b9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.258023 20384 raft_consensus.cc:374] T 29f4ee03917646f6a5374217bed7a1be P fceaf96557de46c3986c306fa4ac3b9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:33.258205 20384 raft_consensus.cc:729] T 29f4ee03917646f6a5374217bed7a1be P fceaf96557de46c3986c306fa4ac3b9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fceaf96557de46c3986c306fa4ac3b9e, State: Initialized, Role: FOLLOWER
I20240503 14:13:33.258936 20382 raft_consensus.cc:482] T c45ceab3a36d4225aed74654ed530c9d P a5d66df07715448fab64928f5bc8299a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:33.259086 20382 raft_consensus.cc:504] T c45ceab3a36d4225aed74654ed530c9d P a5d66df07715448fab64928f5bc8299a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.260062 20334 raft_consensus.cc:2454] T 904450ab5e8b4250b7af18b98d2b0985 P fceaf96557de46c3986c306fa4ac3b9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5d66df07715448fab64928f5bc8299a in term 1.
I20240503 14:13:33.260481 20226 leader_election.cc:302] T 904450ab5e8b4250b7af18b98d2b0985 P a5d66df07715448fab64928f5bc8299a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a5d66df07715448fab64928f5bc8299a, fceaf96557de46c3986c306fa4ac3b9e; no voters: 64ad8799c8df4b7589869baffb93b68d
I20240503 14:13:33.260787 20409 raft_consensus.cc:2781] T 904450ab5e8b4250b7af18b98d2b0985 P a5d66df07715448fab64928f5bc8299a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:33.260943 20409 raft_consensus.cc:686] T 904450ab5e8b4250b7af18b98d2b0985 P a5d66df07715448fab64928f5bc8299a [term 1 LEADER]: Becoming Leader. State: Replica: a5d66df07715448fab64928f5bc8299a, State: Running, Role: LEADER
I20240503 14:13:33.261423 20382 leader_election.cc:288] T c45ceab3a36d4225aed74654ed530c9d P a5d66df07715448fab64928f5bc8299a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64ad8799c8df4b7589869baffb93b68d (127.16.94.129:37429), fceaf96557de46c3986c306fa4ac3b9e (127.16.94.131:39581)
I20240503 14:13:33.261377 20185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c45ceab3a36d4225aed74654ed530c9d" candidate_uuid: "a5d66df07715448fab64928f5bc8299a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64ad8799c8df4b7589869baffb93b68d" is_pre_election: true
I20240503 14:13:33.261286 20409 consensus_queue.cc:237] T 904450ab5e8b4250b7af18b98d2b0985 P a5d66df07715448fab64928f5bc8299a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.259075 20384 consensus_queue.cc:260] T 29f4ee03917646f6a5374217bed7a1be P fceaf96557de46c3986c306fa4ac3b9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
W20240503 14:13:33.261845 20224 leader_election.cc:341] T c45ceab3a36d4225aed74654ed530c9d P a5d66df07715448fab64928f5bc8299a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 64ad8799c8df4b7589869baffb93b68d (127.16.94.129:37429): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:33.262352 20384 ts_tablet_manager.cc:1398] T 29f4ee03917646f6a5374217bed7a1be P fceaf96557de46c3986c306fa4ac3b9e: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20240503 14:13:33.262634 20384 tablet_bootstrap.cc:492] T 44a4d2b820fc4d0da06c659fe98fe57c P fceaf96557de46c3986c306fa4ac3b9e: Bootstrap starting.
I20240503 14:13:33.262758 20334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c45ceab3a36d4225aed74654ed530c9d" candidate_uuid: "a5d66df07715448fab64928f5bc8299a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fceaf96557de46c3986c306fa4ac3b9e" is_pre_election: true
I20240503 14:13:33.262959 20334 raft_consensus.cc:2454] T c45ceab3a36d4225aed74654ed530c9d P fceaf96557de46c3986c306fa4ac3b9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5d66df07715448fab64928f5bc8299a in term 0.
I20240503 14:13:33.263260 20226 leader_election.cc:302] T c45ceab3a36d4225aed74654ed530c9d P a5d66df07715448fab64928f5bc8299a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a5d66df07715448fab64928f5bc8299a, fceaf96557de46c3986c306fa4ac3b9e; no voters: 64ad8799c8df4b7589869baffb93b68d
I20240503 14:13:33.263684 20408 raft_consensus.cc:2781] T c45ceab3a36d4225aed74654ed530c9d P a5d66df07715448fab64928f5bc8299a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:33.263803 20408 raft_consensus.cc:482] T c45ceab3a36d4225aed74654ed530c9d P a5d66df07715448fab64928f5bc8299a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:33.263887 20408 raft_consensus.cc:3037] T c45ceab3a36d4225aed74654ed530c9d P a5d66df07715448fab64928f5bc8299a [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:33.264485 20384 tablet_bootstrap.cc:654] T 44a4d2b820fc4d0da06c659fe98fe57c P fceaf96557de46c3986c306fa4ac3b9e: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:33.267750 20408 raft_consensus.cc:504] T c45ceab3a36d4225aed74654ed530c9d P a5d66df07715448fab64928f5bc8299a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.269013 20185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c45ceab3a36d4225aed74654ed530c9d" candidate_uuid: "a5d66df07715448fab64928f5bc8299a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64ad8799c8df4b7589869baffb93b68d"
W20240503 14:13:33.269565 20224 leader_election.cc:341] T c45ceab3a36d4225aed74654ed530c9d P a5d66df07715448fab64928f5bc8299a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 64ad8799c8df4b7589869baffb93b68d (127.16.94.129:37429): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:33.270123 20383 tablet_bootstrap.cc:492] T 3054a11a8da4470ea5241b92d164d6a8 P 64ad8799c8df4b7589869baffb93b68d: No bootstrap required, opened a new log
I20240503 14:13:33.270280 20383 ts_tablet_manager.cc:1367] T 3054a11a8da4470ea5241b92d164d6a8 P 64ad8799c8df4b7589869baffb93b68d: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.009s
I20240503 14:13:33.270478 20334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c45ceab3a36d4225aed74654ed530c9d" candidate_uuid: "a5d66df07715448fab64928f5bc8299a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fceaf96557de46c3986c306fa4ac3b9e"
I20240503 14:13:33.270673 20334 raft_consensus.cc:3037] T c45ceab3a36d4225aed74654ed530c9d P fceaf96557de46c3986c306fa4ac3b9e [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:33.271176 20383 raft_consensus.cc:348] T 3054a11a8da4470ea5241b92d164d6a8 P 64ad8799c8df4b7589869baffb93b68d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.271363 20383 raft_consensus.cc:374] T 3054a11a8da4470ea5241b92d164d6a8 P 64ad8799c8df4b7589869baffb93b68d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:33.271440 20383 raft_consensus.cc:729] T 3054a11a8da4470ea5241b92d164d6a8 P 64ad8799c8df4b7589869baffb93b68d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64ad8799c8df4b7589869baffb93b68d, State: Initialized, Role: FOLLOWER
I20240503 14:13:33.271687 20383 consensus_queue.cc:260] T 3054a11a8da4470ea5241b92d164d6a8 P 64ad8799c8df4b7589869baffb93b68d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.272755 20082 catalog_manager.cc:5484] T 904450ab5e8b4250b7af18b98d2b0985 P a5d66df07715448fab64928f5bc8299a reported cstate change: term changed from 0 to 1, leader changed from <none> to a5d66df07715448fab64928f5bc8299a (127.16.94.130). New cstate: current_term: 1 leader_uuid: "a5d66df07715448fab64928f5bc8299a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:33.273949 20334 raft_consensus.cc:2454] T c45ceab3a36d4225aed74654ed530c9d P fceaf96557de46c3986c306fa4ac3b9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5d66df07715448fab64928f5bc8299a in term 1.
I20240503 14:13:33.274448 20383 ts_tablet_manager.cc:1398] T 3054a11a8da4470ea5241b92d164d6a8 P 64ad8799c8df4b7589869baffb93b68d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:13:33.274574 20226 leader_election.cc:302] T c45ceab3a36d4225aed74654ed530c9d P a5d66df07715448fab64928f5bc8299a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a5d66df07715448fab64928f5bc8299a, fceaf96557de46c3986c306fa4ac3b9e; no voters: 64ad8799c8df4b7589869baffb93b68d
I20240503 14:13:33.274935 20409 raft_consensus.cc:2781] T c45ceab3a36d4225aed74654ed530c9d P a5d66df07715448fab64928f5bc8299a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:33.275082 20409 raft_consensus.cc:686] T c45ceab3a36d4225aed74654ed530c9d P a5d66df07715448fab64928f5bc8299a [term 1 LEADER]: Becoming Leader. State: Replica: a5d66df07715448fab64928f5bc8299a, State: Running, Role: LEADER
I20240503 14:13:33.275346 20409 consensus_queue.cc:237] T c45ceab3a36d4225aed74654ed530c9d P a5d66df07715448fab64928f5bc8299a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.276902 20383 tablet_bootstrap.cc:492] T 7a82ae1f4c66455caea352a304af2b51 P 64ad8799c8df4b7589869baffb93b68d: Bootstrap starting.
I20240503 14:13:33.278498 20408 leader_election.cc:288] T c45ceab3a36d4225aed74654ed530c9d P a5d66df07715448fab64928f5bc8299a [CANDIDATE]: Term 1 election: Requested vote from peers 64ad8799c8df4b7589869baffb93b68d (127.16.94.129:37429), fceaf96557de46c3986c306fa4ac3b9e (127.16.94.131:39581)
I20240503 14:13:33.279069 20383 tablet_bootstrap.cc:654] T 7a82ae1f4c66455caea352a304af2b51 P 64ad8799c8df4b7589869baffb93b68d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:33.279889 20380 tablet_bootstrap.cc:492] T d4743d5a73c54086b277f4c4b1a248fd P a5d66df07715448fab64928f5bc8299a: No bootstrap required, opened a new log
I20240503 14:13:33.280164 20380 ts_tablet_manager.cc:1367] T d4743d5a73c54086b277f4c4b1a248fd P a5d66df07715448fab64928f5bc8299a: Time spent bootstrapping tablet: real 0.050s user 0.015s sys 0.002s
I20240503 14:13:33.281272 20380 raft_consensus.cc:348] T d4743d5a73c54086b277f4c4b1a248fd P a5d66df07715448fab64928f5bc8299a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.281687 20380 raft_consensus.cc:374] T d4743d5a73c54086b277f4c4b1a248fd P a5d66df07715448fab64928f5bc8299a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:33.281857 20380 raft_consensus.cc:729] T d4743d5a73c54086b277f4c4b1a248fd P a5d66df07715448fab64928f5bc8299a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5d66df07715448fab64928f5bc8299a, State: Initialized, Role: FOLLOWER
I20240503 14:13:33.282291 20380 consensus_queue.cc:260] T d4743d5a73c54086b277f4c4b1a248fd P a5d66df07715448fab64928f5bc8299a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.283288 20380 ts_tablet_manager.cc:1398] T d4743d5a73c54086b277f4c4b1a248fd P a5d66df07715448fab64928f5bc8299a: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 14:13:33.283749 20380 tablet_bootstrap.cc:492] T 0eb7552695304a1ea6f8e02f189f7295 P a5d66df07715448fab64928f5bc8299a: Bootstrap starting.
I20240503 14:13:33.284050 20388 raft_consensus.cc:482] T 0eb7552695304a1ea6f8e02f189f7295 P fceaf96557de46c3986c306fa4ac3b9e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:33.284263 20388 raft_consensus.cc:504] T 0eb7552695304a1ea6f8e02f189f7295 P fceaf96557de46c3986c306fa4ac3b9e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.285141 20388 leader_election.cc:288] T 0eb7552695304a1ea6f8e02f189f7295 P fceaf96557de46c3986c306fa4ac3b9e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64ad8799c8df4b7589869baffb93b68d (127.16.94.129:37429), a5d66df07715448fab64928f5bc8299a (127.16.94.130:39295)
I20240503 14:13:33.285950 20185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eb7552695304a1ea6f8e02f189f7295" candidate_uuid: "fceaf96557de46c3986c306fa4ac3b9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64ad8799c8df4b7589869baffb93b68d" is_pre_election: true
I20240503 14:13:33.286216 20185 raft_consensus.cc:2454] T 0eb7552695304a1ea6f8e02f189f7295 P 64ad8799c8df4b7589869baffb93b68d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fceaf96557de46c3986c306fa4ac3b9e in term 0.
I20240503 14:13:33.286844 20298 leader_election.cc:302] T 0eb7552695304a1ea6f8e02f189f7295 P fceaf96557de46c3986c306fa4ac3b9e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64ad8799c8df4b7589869baffb93b68d, fceaf96557de46c3986c306fa4ac3b9e; no voters:
I20240503 14:13:33.286226 20261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eb7552695304a1ea6f8e02f189f7295" candidate_uuid: "fceaf96557de46c3986c306fa4ac3b9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5d66df07715448fab64928f5bc8299a" is_pre_election: true
W20240503 14:13:33.287812 20301 leader_election.cc:341] T 0eb7552695304a1ea6f8e02f189f7295 P fceaf96557de46c3986c306fa4ac3b9e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a5d66df07715448fab64928f5bc8299a (127.16.94.130:39295): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:33.288336 20388 raft_consensus.cc:2781] T 0eb7552695304a1ea6f8e02f189f7295 P fceaf96557de46c3986c306fa4ac3b9e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:33.288491 20388 raft_consensus.cc:482] T 0eb7552695304a1ea6f8e02f189f7295 P fceaf96557de46c3986c306fa4ac3b9e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:33.288604 20388 raft_consensus.cc:3037] T 0eb7552695304a1ea6f8e02f189f7295 P fceaf96557de46c3986c306fa4ac3b9e [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:33.291889 20082 catalog_manager.cc:5484] T c45ceab3a36d4225aed74654ed530c9d P a5d66df07715448fab64928f5bc8299a reported cstate change: term changed from 0 to 1, leader changed from <none> to a5d66df07715448fab64928f5bc8299a (127.16.94.130). New cstate: current_term: 1 leader_uuid: "a5d66df07715448fab64928f5bc8299a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:33.292654 20388 raft_consensus.cc:504] T 0eb7552695304a1ea6f8e02f189f7295 P fceaf96557de46c3986c306fa4ac3b9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.294047 20185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eb7552695304a1ea6f8e02f189f7295" candidate_uuid: "fceaf96557de46c3986c306fa4ac3b9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64ad8799c8df4b7589869baffb93b68d"
I20240503 14:13:33.294298 20185 raft_consensus.cc:3037] T 0eb7552695304a1ea6f8e02f189f7295 P 64ad8799c8df4b7589869baffb93b68d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:33.298287 20185 raft_consensus.cc:2454] T 0eb7552695304a1ea6f8e02f189f7295 P 64ad8799c8df4b7589869baffb93b68d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fceaf96557de46c3986c306fa4ac3b9e in term 1.
I20240503 14:13:33.298859 20298 leader_election.cc:302] T 0eb7552695304a1ea6f8e02f189f7295 P fceaf96557de46c3986c306fa4ac3b9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64ad8799c8df4b7589869baffb93b68d, fceaf96557de46c3986c306fa4ac3b9e; no voters:
I20240503 14:13:33.299218 20420 raft_consensus.cc:2781] T 0eb7552695304a1ea6f8e02f189f7295 P fceaf96557de46c3986c306fa4ac3b9e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:33.299770 20420 raft_consensus.cc:686] T 0eb7552695304a1ea6f8e02f189f7295 P fceaf96557de46c3986c306fa4ac3b9e [term 1 LEADER]: Becoming Leader. State: Replica: fceaf96557de46c3986c306fa4ac3b9e, State: Running, Role: LEADER
I20240503 14:13:33.300125 20420 consensus_queue.cc:237] T 0eb7552695304a1ea6f8e02f189f7295 P fceaf96557de46c3986c306fa4ac3b9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.302354 20261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eb7552695304a1ea6f8e02f189f7295" candidate_uuid: "fceaf96557de46c3986c306fa4ac3b9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5d66df07715448fab64928f5bc8299a"
W20240503 14:13:33.303045 20301 leader_election.cc:341] T 0eb7552695304a1ea6f8e02f189f7295 P fceaf96557de46c3986c306fa4ac3b9e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a5d66df07715448fab64928f5bc8299a (127.16.94.130:39295): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:33.303326 20388 leader_election.cc:288] T 0eb7552695304a1ea6f8e02f189f7295 P fceaf96557de46c3986c306fa4ac3b9e [CANDIDATE]: Term 1 election: Requested vote from peers 64ad8799c8df4b7589869baffb93b68d (127.16.94.129:37429), a5d66df07715448fab64928f5bc8299a (127.16.94.130:39295)
I20240503 14:13:33.303812 20389 raft_consensus.cc:482] T 29f4ee03917646f6a5374217bed7a1be P 64ad8799c8df4b7589869baffb93b68d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:33.303689 20082 catalog_manager.cc:5484] T 0eb7552695304a1ea6f8e02f189f7295 P fceaf96557de46c3986c306fa4ac3b9e reported cstate change: term changed from 0 to 1, leader changed from <none> to fceaf96557de46c3986c306fa4ac3b9e (127.16.94.131). New cstate: current_term: 1 leader_uuid: "fceaf96557de46c3986c306fa4ac3b9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } health_report { overall_health: HEALTHY } } }
I20240503 14:13:33.303946 20389 raft_consensus.cc:504] T 29f4ee03917646f6a5374217bed7a1be P 64ad8799c8df4b7589869baffb93b68d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.304474 20380 tablet_bootstrap.cc:654] T 0eb7552695304a1ea6f8e02f189f7295 P a5d66df07715448fab64928f5bc8299a: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:33.304509 20389 leader_election.cc:288] T 29f4ee03917646f6a5374217bed7a1be P 64ad8799c8df4b7589869baffb93b68d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5d66df07715448fab64928f5bc8299a (127.16.94.130:39295), fceaf96557de46c3986c306fa4ac3b9e (127.16.94.131:39581)
I20240503 14:13:33.305794 20334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29f4ee03917646f6a5374217bed7a1be" candidate_uuid: "64ad8799c8df4b7589869baffb93b68d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fceaf96557de46c3986c306fa4ac3b9e" is_pre_election: true
I20240503 14:13:33.305999 20261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29f4ee03917646f6a5374217bed7a1be" candidate_uuid: "64ad8799c8df4b7589869baffb93b68d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5d66df07715448fab64928f5bc8299a" is_pre_election: true
I20240503 14:13:33.306262 20261 raft_consensus.cc:2454] T 29f4ee03917646f6a5374217bed7a1be P a5d66df07715448fab64928f5bc8299a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64ad8799c8df4b7589869baffb93b68d in term 0.
I20240503 14:13:33.306782 20151 leader_election.cc:302] T 29f4ee03917646f6a5374217bed7a1be P 64ad8799c8df4b7589869baffb93b68d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64ad8799c8df4b7589869baffb93b68d, a5d66df07715448fab64928f5bc8299a; no voters:
I20240503 14:13:33.306062 20334 raft_consensus.cc:2454] T 29f4ee03917646f6a5374217bed7a1be P fceaf96557de46c3986c306fa4ac3b9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64ad8799c8df4b7589869baffb93b68d in term 0.
I20240503 14:13:33.307082 20389 raft_consensus.cc:2781] T 29f4ee03917646f6a5374217bed7a1be P 64ad8799c8df4b7589869baffb93b68d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:33.307176 20389 raft_consensus.cc:482] T 29f4ee03917646f6a5374217bed7a1be P 64ad8799c8df4b7589869baffb93b68d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:33.307263 20389 raft_consensus.cc:3037] T 29f4ee03917646f6a5374217bed7a1be P 64ad8799c8df4b7589869baffb93b68d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:33.309456 20409 raft_consensus.cc:482] T d4743d5a73c54086b277f4c4b1a248fd P a5d66df07715448fab64928f5bc8299a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:33.309835 20409 raft_consensus.cc:504] T d4743d5a73c54086b277f4c4b1a248fd P a5d66df07715448fab64928f5bc8299a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.312585 20389 raft_consensus.cc:504] T 29f4ee03917646f6a5374217bed7a1be P 64ad8799c8df4b7589869baffb93b68d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.313613 20389 leader_election.cc:288] T 29f4ee03917646f6a5374217bed7a1be P 64ad8799c8df4b7589869baffb93b68d [CANDIDATE]: Term 1 election: Requested vote from peers a5d66df07715448fab64928f5bc8299a (127.16.94.130:39295), fceaf96557de46c3986c306fa4ac3b9e (127.16.94.131:39581)
I20240503 14:13:33.313777 20261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29f4ee03917646f6a5374217bed7a1be" candidate_uuid: "64ad8799c8df4b7589869baffb93b68d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5d66df07715448fab64928f5bc8299a"
I20240503 14:13:33.313849 20334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29f4ee03917646f6a5374217bed7a1be" candidate_uuid: "64ad8799c8df4b7589869baffb93b68d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fceaf96557de46c3986c306fa4ac3b9e"
I20240503 14:13:33.313993 20334 raft_consensus.cc:3037] T 29f4ee03917646f6a5374217bed7a1be P fceaf96557de46c3986c306fa4ac3b9e [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:33.313992 20261 raft_consensus.cc:3037] T 29f4ee03917646f6a5374217bed7a1be P a5d66df07715448fab64928f5bc8299a [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:33.316288 20384 tablet_bootstrap.cc:492] T 44a4d2b820fc4d0da06c659fe98fe57c P fceaf96557de46c3986c306fa4ac3b9e: No bootstrap required, opened a new log
I20240503 14:13:33.316540 20384 ts_tablet_manager.cc:1367] T 44a4d2b820fc4d0da06c659fe98fe57c P fceaf96557de46c3986c306fa4ac3b9e: Time spent bootstrapping tablet: real 0.054s user 0.015s sys 0.030s
I20240503 14:13:33.317354 20334 raft_consensus.cc:2454] T 29f4ee03917646f6a5374217bed7a1be P fceaf96557de46c3986c306fa4ac3b9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64ad8799c8df4b7589869baffb93b68d in term 1.
I20240503 14:13:33.317354 20261 raft_consensus.cc:2454] T 29f4ee03917646f6a5374217bed7a1be P a5d66df07715448fab64928f5bc8299a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64ad8799c8df4b7589869baffb93b68d in term 1.
I20240503 14:13:33.317628 20335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4743d5a73c54086b277f4c4b1a248fd" candidate_uuid: "a5d66df07715448fab64928f5bc8299a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fceaf96557de46c3986c306fa4ac3b9e" is_pre_election: true
I20240503 14:13:33.317607 20384 raft_consensus.cc:348] T 44a4d2b820fc4d0da06c659fe98fe57c P fceaf96557de46c3986c306fa4ac3b9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.317909 20384 raft_consensus.cc:374] T 44a4d2b820fc4d0da06c659fe98fe57c P fceaf96557de46c3986c306fa4ac3b9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:33.317958 20409 leader_election.cc:288] T d4743d5a73c54086b277f4c4b1a248fd P a5d66df07715448fab64928f5bc8299a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64ad8799c8df4b7589869baffb93b68d (127.16.94.129:37429), fceaf96557de46c3986c306fa4ac3b9e (127.16.94.131:39581)
I20240503 14:13:33.318032 20384 raft_consensus.cc:729] T 44a4d2b820fc4d0da06c659fe98fe57c P fceaf96557de46c3986c306fa4ac3b9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fceaf96557de46c3986c306fa4ac3b9e, State: Initialized, Role: FOLLOWER
W20240503 14:13:33.318224 20226 leader_election.cc:341] T d4743d5a73c54086b277f4c4b1a248fd P a5d66df07715448fab64928f5bc8299a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fceaf96557de46c3986c306fa4ac3b9e (127.16.94.131:39581): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:33.318763 20185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4743d5a73c54086b277f4c4b1a248fd" candidate_uuid: "a5d66df07715448fab64928f5bc8299a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64ad8799c8df4b7589869baffb93b68d" is_pre_election: true
I20240503 14:13:33.318769 20151 leader_election.cc:302] T 29f4ee03917646f6a5374217bed7a1be P 64ad8799c8df4b7589869baffb93b68d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64ad8799c8df4b7589869baffb93b68d, a5d66df07715448fab64928f5bc8299a; no voters:
I20240503 14:13:33.318979 20185 raft_consensus.cc:2454] T d4743d5a73c54086b277f4c4b1a248fd P 64ad8799c8df4b7589869baffb93b68d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5d66df07715448fab64928f5bc8299a in term 0.
I20240503 14:13:33.318320 20384 consensus_queue.cc:260] T 44a4d2b820fc4d0da06c659fe98fe57c P fceaf96557de46c3986c306fa4ac3b9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.319147 20389 raft_consensus.cc:2781] T 29f4ee03917646f6a5374217bed7a1be P 64ad8799c8df4b7589869baffb93b68d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:33.319350 20224 leader_election.cc:302] T d4743d5a73c54086b277f4c4b1a248fd P a5d66df07715448fab64928f5bc8299a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 64ad8799c8df4b7589869baffb93b68d, a5d66df07715448fab64928f5bc8299a; no voters: fceaf96557de46c3986c306fa4ac3b9e
I20240503 14:13:33.319600 20384 ts_tablet_manager.cc:1398] T 44a4d2b820fc4d0da06c659fe98fe57c P fceaf96557de46c3986c306fa4ac3b9e: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 14:13:33.319933 20384 tablet_bootstrap.cc:492] T d4743d5a73c54086b277f4c4b1a248fd P fceaf96557de46c3986c306fa4ac3b9e: Bootstrap starting.
I20240503 14:13:33.320034 20409 raft_consensus.cc:2781] T d4743d5a73c54086b277f4c4b1a248fd P a5d66df07715448fab64928f5bc8299a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:33.320142 20409 raft_consensus.cc:482] T d4743d5a73c54086b277f4c4b1a248fd P a5d66df07715448fab64928f5bc8299a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:33.320298 20409 raft_consensus.cc:3037] T d4743d5a73c54086b277f4c4b1a248fd P a5d66df07715448fab64928f5bc8299a [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:33.320694 20383 tablet_bootstrap.cc:492] T 7a82ae1f4c66455caea352a304af2b51 P 64ad8799c8df4b7589869baffb93b68d: No bootstrap required, opened a new log
I20240503 14:13:33.320919 20383 ts_tablet_manager.cc:1367] T 7a82ae1f4c66455caea352a304af2b51 P 64ad8799c8df4b7589869baffb93b68d: Time spent bootstrapping tablet: real 0.044s user 0.019s sys 0.020s
I20240503 14:13:33.321770 20383 raft_consensus.cc:348] T 7a82ae1f4c66455caea352a304af2b51 P 64ad8799c8df4b7589869baffb93b68d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.321938 20384 tablet_bootstrap.cc:654] T d4743d5a73c54086b277f4c4b1a248fd P fceaf96557de46c3986c306fa4ac3b9e: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:33.322039 20383 raft_consensus.cc:374] T 7a82ae1f4c66455caea352a304af2b51 P 64ad8799c8df4b7589869baffb93b68d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:33.322161 20383 raft_consensus.cc:729] T 7a82ae1f4c66455caea352a304af2b51 P 64ad8799c8df4b7589869baffb93b68d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64ad8799c8df4b7589869baffb93b68d, State: Initialized, Role: FOLLOWER
I20240503 14:13:33.322427 20383 consensus_queue.cc:260] T 7a82ae1f4c66455caea352a304af2b51 P 64ad8799c8df4b7589869baffb93b68d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.323060 20383 ts_tablet_manager.cc:1398] T 7a82ae1f4c66455caea352a304af2b51 P 64ad8799c8df4b7589869baffb93b68d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:33.323415 20383 tablet_bootstrap.cc:492] T 904450ab5e8b4250b7af18b98d2b0985 P 64ad8799c8df4b7589869baffb93b68d: Bootstrap starting.
I20240503 14:13:33.324432 20389 raft_consensus.cc:686] T 29f4ee03917646f6a5374217bed7a1be P 64ad8799c8df4b7589869baffb93b68d [term 1 LEADER]: Becoming Leader. State: Replica: 64ad8799c8df4b7589869baffb93b68d, State: Running, Role: LEADER
I20240503 14:13:33.324765 20389 consensus_queue.cc:237] T 29f4ee03917646f6a5374217bed7a1be P 64ad8799c8df4b7589869baffb93b68d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.325242 20383 tablet_bootstrap.cc:654] T 904450ab5e8b4250b7af18b98d2b0985 P 64ad8799c8df4b7589869baffb93b68d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:33.328738 20409 raft_consensus.cc:504] T d4743d5a73c54086b277f4c4b1a248fd P a5d66df07715448fab64928f5bc8299a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.330327 20185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4743d5a73c54086b277f4c4b1a248fd" candidate_uuid: "a5d66df07715448fab64928f5bc8299a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64ad8799c8df4b7589869baffb93b68d"
I20240503 14:13:33.330569 20185 raft_consensus.cc:3037] T d4743d5a73c54086b277f4c4b1a248fd P 64ad8799c8df4b7589869baffb93b68d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:33.343926 20185 raft_consensus.cc:2454] T d4743d5a73c54086b277f4c4b1a248fd P 64ad8799c8df4b7589869baffb93b68d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5d66df07715448fab64928f5bc8299a in term 1.
I20240503 14:13:33.344825 20224 leader_election.cc:302] T d4743d5a73c54086b277f4c4b1a248fd P a5d66df07715448fab64928f5bc8299a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64ad8799c8df4b7589869baffb93b68d, a5d66df07715448fab64928f5bc8299a; no voters:
I20240503 14:13:33.345882 20408 raft_consensus.cc:2781] T d4743d5a73c54086b277f4c4b1a248fd P a5d66df07715448fab64928f5bc8299a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:33.346060 20408 raft_consensus.cc:686] T d4743d5a73c54086b277f4c4b1a248fd P a5d66df07715448fab64928f5bc8299a [term 1 LEADER]: Becoming Leader. State: Replica: a5d66df07715448fab64928f5bc8299a, State: Running, Role: LEADER
I20240503 14:13:33.346433 20408 consensus_queue.cc:237] T d4743d5a73c54086b277f4c4b1a248fd P a5d66df07715448fab64928f5bc8299a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.347889 20382 raft_consensus.cc:482] T 3054a11a8da4470ea5241b92d164d6a8 P a5d66df07715448fab64928f5bc8299a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:33.348073 20382 raft_consensus.cc:504] T 3054a11a8da4470ea5241b92d164d6a8 P a5d66df07715448fab64928f5bc8299a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.348937 20382 leader_election.cc:288] T 3054a11a8da4470ea5241b92d164d6a8 P a5d66df07715448fab64928f5bc8299a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64ad8799c8df4b7589869baffb93b68d (127.16.94.129:37429), fceaf96557de46c3986c306fa4ac3b9e (127.16.94.131:39581)
I20240503 14:13:33.350598 20335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3054a11a8da4470ea5241b92d164d6a8" candidate_uuid: "a5d66df07715448fab64928f5bc8299a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fceaf96557de46c3986c306fa4ac3b9e" is_pre_election: true
I20240503 14:13:33.350862 20335 raft_consensus.cc:2454] T 3054a11a8da4470ea5241b92d164d6a8 P fceaf96557de46c3986c306fa4ac3b9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5d66df07715448fab64928f5bc8299a in term 0.
I20240503 14:13:33.352321 20226 leader_election.cc:302] T 3054a11a8da4470ea5241b92d164d6a8 P a5d66df07715448fab64928f5bc8299a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5d66df07715448fab64928f5bc8299a, fceaf96557de46c3986c306fa4ac3b9e; no voters:
I20240503 14:13:33.352619 20437 raft_consensus.cc:2781] T 3054a11a8da4470ea5241b92d164d6a8 P a5d66df07715448fab64928f5bc8299a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:33.352829 20437 raft_consensus.cc:482] T 3054a11a8da4470ea5241b92d164d6a8 P a5d66df07715448fab64928f5bc8299a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:33.353034 20437 raft_consensus.cc:3037] T 3054a11a8da4470ea5241b92d164d6a8 P a5d66df07715448fab64928f5bc8299a [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:33.322647 20380 tablet_bootstrap.cc:492] T 0eb7552695304a1ea6f8e02f189f7295 P a5d66df07715448fab64928f5bc8299a: No bootstrap required, opened a new log
I20240503 14:13:33.354558 20184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3054a11a8da4470ea5241b92d164d6a8" candidate_uuid: "a5d66df07715448fab64928f5bc8299a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64ad8799c8df4b7589869baffb93b68d" is_pre_election: true
I20240503 14:13:33.354863 20184 raft_consensus.cc:2454] T 3054a11a8da4470ea5241b92d164d6a8 P 64ad8799c8df4b7589869baffb93b68d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5d66df07715448fab64928f5bc8299a in term 0.
I20240503 14:13:33.356289 20335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4743d5a73c54086b277f4c4b1a248fd" candidate_uuid: "a5d66df07715448fab64928f5bc8299a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fceaf96557de46c3986c306fa4ac3b9e"
W20240503 14:13:33.357175 20226 leader_election.cc:341] T d4743d5a73c54086b277f4c4b1a248fd P a5d66df07715448fab64928f5bc8299a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fceaf96557de46c3986c306fa4ac3b9e (127.16.94.131:39581): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:33.358753 20081 catalog_manager.cc:5484] T d4743d5a73c54086b277f4c4b1a248fd P a5d66df07715448fab64928f5bc8299a reported cstate change: term changed from 0 to 1, leader changed from <none> to a5d66df07715448fab64928f5bc8299a (127.16.94.130). New cstate: current_term: 1 leader_uuid: "a5d66df07715448fab64928f5bc8299a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:33.360100 20380 ts_tablet_manager.cc:1367] T 0eb7552695304a1ea6f8e02f189f7295 P a5d66df07715448fab64928f5bc8299a: Time spent bootstrapping tablet: real 0.070s user 0.005s sys 0.001s
I20240503 14:13:33.362736 20380 raft_consensus.cc:348] T 0eb7552695304a1ea6f8e02f189f7295 P a5d66df07715448fab64928f5bc8299a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.363021 20380 raft_consensus.cc:374] T 0eb7552695304a1ea6f8e02f189f7295 P a5d66df07715448fab64928f5bc8299a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:33.363118 20380 raft_consensus.cc:729] T 0eb7552695304a1ea6f8e02f189f7295 P a5d66df07715448fab64928f5bc8299a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5d66df07715448fab64928f5bc8299a, State: Initialized, Role: FOLLOWER
I20240503 14:13:33.363461 20380 consensus_queue.cc:260] T 0eb7552695304a1ea6f8e02f189f7295 P a5d66df07715448fab64928f5bc8299a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.364342 20380 ts_tablet_manager.cc:1398] T 0eb7552695304a1ea6f8e02f189f7295 P a5d66df07715448fab64928f5bc8299a: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:13:33.368207 20380 tablet_bootstrap.cc:492] T 7a82ae1f4c66455caea352a304af2b51 P a5d66df07715448fab64928f5bc8299a: Bootstrap starting.
I20240503 14:13:33.369737 20409 leader_election.cc:288] T d4743d5a73c54086b277f4c4b1a248fd P a5d66df07715448fab64928f5bc8299a [CANDIDATE]: Term 1 election: Requested vote from peers 64ad8799c8df4b7589869baffb93b68d (127.16.94.129:37429), fceaf96557de46c3986c306fa4ac3b9e (127.16.94.131:39581)
I20240503 14:13:33.371521 20380 tablet_bootstrap.cc:654] T 7a82ae1f4c66455caea352a304af2b51 P a5d66df07715448fab64928f5bc8299a: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:33.379359 20384 tablet_bootstrap.cc:492] T d4743d5a73c54086b277f4c4b1a248fd P fceaf96557de46c3986c306fa4ac3b9e: No bootstrap required, opened a new log
I20240503 14:13:33.379712 20384 ts_tablet_manager.cc:1367] T d4743d5a73c54086b277f4c4b1a248fd P fceaf96557de46c3986c306fa4ac3b9e: Time spent bootstrapping tablet: real 0.060s user 0.022s sys 0.031s
I20240503 14:13:33.380416 20383 tablet_bootstrap.cc:492] T 904450ab5e8b4250b7af18b98d2b0985 P 64ad8799c8df4b7589869baffb93b68d: No bootstrap required, opened a new log
I20240503 14:13:33.380609 20383 ts_tablet_manager.cc:1367] T 904450ab5e8b4250b7af18b98d2b0985 P 64ad8799c8df4b7589869baffb93b68d: Time spent bootstrapping tablet: real 0.057s user 0.018s sys 0.032s
I20240503 14:13:33.380820 20081 catalog_manager.cc:5484] T 29f4ee03917646f6a5374217bed7a1be P 64ad8799c8df4b7589869baffb93b68d reported cstate change: term changed from 0 to 1, leader changed from <none> to 64ad8799c8df4b7589869baffb93b68d (127.16.94.129). New cstate: current_term: 1 leader_uuid: "64ad8799c8df4b7589869baffb93b68d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:33.380962 20384 raft_consensus.cc:348] T d4743d5a73c54086b277f4c4b1a248fd P fceaf96557de46c3986c306fa4ac3b9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.381318 20383 raft_consensus.cc:348] T 904450ab5e8b4250b7af18b98d2b0985 P 64ad8799c8df4b7589869baffb93b68d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.381429 20384 raft_consensus.cc:374] T d4743d5a73c54086b277f4c4b1a248fd P fceaf96557de46c3986c306fa4ac3b9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:33.381518 20383 raft_consensus.cc:374] T 904450ab5e8b4250b7af18b98d2b0985 P 64ad8799c8df4b7589869baffb93b68d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:33.381651 20384 raft_consensus.cc:729] T d4743d5a73c54086b277f4c4b1a248fd P fceaf96557de46c3986c306fa4ac3b9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fceaf96557de46c3986c306fa4ac3b9e, State: Initialized, Role: FOLLOWER
I20240503 14:13:33.381808 20383 raft_consensus.cc:729] T 904450ab5e8b4250b7af18b98d2b0985 P 64ad8799c8df4b7589869baffb93b68d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64ad8799c8df4b7589869baffb93b68d, State: Initialized, Role: FOLLOWER
I20240503 14:13:33.382154 20383 consensus_queue.cc:260] T 904450ab5e8b4250b7af18b98d2b0985 P 64ad8799c8df4b7589869baffb93b68d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.382129 20384 consensus_queue.cc:260] T d4743d5a73c54086b277f4c4b1a248fd P fceaf96557de46c3986c306fa4ac3b9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.382756 20383 ts_tablet_manager.cc:1398] T 904450ab5e8b4250b7af18b98d2b0985 P 64ad8799c8df4b7589869baffb93b68d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:13:33.383045 20380 tablet_bootstrap.cc:492] T 7a82ae1f4c66455caea352a304af2b51 P a5d66df07715448fab64928f5bc8299a: No bootstrap required, opened a new log
I20240503 14:13:33.383085 20383 tablet_bootstrap.cc:492] T c45ceab3a36d4225aed74654ed530c9d P 64ad8799c8df4b7589869baffb93b68d: Bootstrap starting.
I20240503 14:13:33.383235 20380 ts_tablet_manager.cc:1367] T 7a82ae1f4c66455caea352a304af2b51 P a5d66df07715448fab64928f5bc8299a: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.008s
I20240503 14:13:33.383934 20384 ts_tablet_manager.cc:1398] T d4743d5a73c54086b277f4c4b1a248fd P fceaf96557de46c3986c306fa4ac3b9e: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 14:13:33.383975 20380 raft_consensus.cc:348] T 7a82ae1f4c66455caea352a304af2b51 P a5d66df07715448fab64928f5bc8299a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.384199 20380 raft_consensus.cc:374] T 7a82ae1f4c66455caea352a304af2b51 P a5d66df07715448fab64928f5bc8299a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:33.384287 20380 raft_consensus.cc:729] T 7a82ae1f4c66455caea352a304af2b51 P a5d66df07715448fab64928f5bc8299a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5d66df07715448fab64928f5bc8299a, State: Initialized, Role: FOLLOWER
I20240503 14:13:33.382302 20437 raft_consensus.cc:504] T 3054a11a8da4470ea5241b92d164d6a8 P a5d66df07715448fab64928f5bc8299a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.384526 20384 tablet_bootstrap.cc:492] T 7905fd813a2d4a8aa8b2ac1e752b37dc P fceaf96557de46c3986c306fa4ac3b9e: Bootstrap starting.
I20240503 14:13:33.384804 20383 tablet_bootstrap.cc:654] T c45ceab3a36d4225aed74654ed530c9d P 64ad8799c8df4b7589869baffb93b68d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:33.385337 20184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3054a11a8da4470ea5241b92d164d6a8" candidate_uuid: "a5d66df07715448fab64928f5bc8299a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64ad8799c8df4b7589869baffb93b68d"
I20240503 14:13:33.385545 20184 raft_consensus.cc:3037] T 3054a11a8da4470ea5241b92d164d6a8 P 64ad8799c8df4b7589869baffb93b68d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:33.386408 20335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3054a11a8da4470ea5241b92d164d6a8" candidate_uuid: "a5d66df07715448fab64928f5bc8299a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fceaf96557de46c3986c306fa4ac3b9e"
I20240503 14:13:33.386615 20335 raft_consensus.cc:3037] T 3054a11a8da4470ea5241b92d164d6a8 P fceaf96557de46c3986c306fa4ac3b9e [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:33.389780 20184 raft_consensus.cc:2454] T 3054a11a8da4470ea5241b92d164d6a8 P 64ad8799c8df4b7589869baffb93b68d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5d66df07715448fab64928f5bc8299a in term 1.
I20240503 14:13:33.390542 20224 leader_election.cc:302] T 3054a11a8da4470ea5241b92d164d6a8 P a5d66df07715448fab64928f5bc8299a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64ad8799c8df4b7589869baffb93b68d, a5d66df07715448fab64928f5bc8299a; no voters:
I20240503 14:13:33.390856 20409 raft_consensus.cc:2781] T 3054a11a8da4470ea5241b92d164d6a8 P a5d66df07715448fab64928f5bc8299a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:33.391000 20409 raft_consensus.cc:686] T 3054a11a8da4470ea5241b92d164d6a8 P a5d66df07715448fab64928f5bc8299a [term 1 LEADER]: Becoming Leader. State: Replica: a5d66df07715448fab64928f5bc8299a, State: Running, Role: LEADER
I20240503 14:13:33.391219 20384 tablet_bootstrap.cc:654] T 7905fd813a2d4a8aa8b2ac1e752b37dc P fceaf96557de46c3986c306fa4ac3b9e: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:33.391283 20409 consensus_queue.cc:237] T 3054a11a8da4470ea5241b92d164d6a8 P a5d66df07715448fab64928f5bc8299a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.389897 20437 leader_election.cc:288] T 3054a11a8da4470ea5241b92d164d6a8 P a5d66df07715448fab64928f5bc8299a [CANDIDATE]: Term 1 election: Requested vote from peers 64ad8799c8df4b7589869baffb93b68d (127.16.94.129:37429), fceaf96557de46c3986c306fa4ac3b9e (127.16.94.131:39581)
I20240503 14:13:33.389807 20335 raft_consensus.cc:2454] T 3054a11a8da4470ea5241b92d164d6a8 P fceaf96557de46c3986c306fa4ac3b9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5d66df07715448fab64928f5bc8299a in term 1.
I20240503 14:13:33.390106 20380 consensus_queue.cc:260] T 7a82ae1f4c66455caea352a304af2b51 P a5d66df07715448fab64928f5bc8299a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.394066 20380 ts_tablet_manager.cc:1398] T 7a82ae1f4c66455caea352a304af2b51 P a5d66df07715448fab64928f5bc8299a: Time spent starting tablet: real 0.011s user 0.001s sys 0.000s
I20240503 14:13:33.393896 20081 catalog_manager.cc:5484] T 3054a11a8da4470ea5241b92d164d6a8 P a5d66df07715448fab64928f5bc8299a reported cstate change: term changed from 0 to 1, leader changed from <none> to a5d66df07715448fab64928f5bc8299a (127.16.94.130). New cstate: current_term: 1 leader_uuid: "a5d66df07715448fab64928f5bc8299a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:33.394477 20380 tablet_bootstrap.cc:492] T ad711d78ee524d979867b65906abe456 P a5d66df07715448fab64928f5bc8299a: Bootstrap starting.
I20240503 14:13:33.396209 20380 tablet_bootstrap.cc:654] T ad711d78ee524d979867b65906abe456 P a5d66df07715448fab64928f5bc8299a: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:33.400316 20383 tablet_bootstrap.cc:492] T c45ceab3a36d4225aed74654ed530c9d P 64ad8799c8df4b7589869baffb93b68d: No bootstrap required, opened a new log
I20240503 14:13:33.400548 20383 ts_tablet_manager.cc:1367] T c45ceab3a36d4225aed74654ed530c9d P 64ad8799c8df4b7589869baffb93b68d: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.013s
I20240503 14:13:33.401001 20380 tablet_bootstrap.cc:492] T ad711d78ee524d979867b65906abe456 P a5d66df07715448fab64928f5bc8299a: No bootstrap required, opened a new log
I20240503 14:13:33.401171 20380 ts_tablet_manager.cc:1367] T ad711d78ee524d979867b65906abe456 P a5d66df07715448fab64928f5bc8299a: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.001s
I20240503 14:13:33.401252 20384 tablet_bootstrap.cc:492] T 7905fd813a2d4a8aa8b2ac1e752b37dc P fceaf96557de46c3986c306fa4ac3b9e: No bootstrap required, opened a new log
I20240503 14:13:33.401454 20384 ts_tablet_manager.cc:1367] T 7905fd813a2d4a8aa8b2ac1e752b37dc P fceaf96557de46c3986c306fa4ac3b9e: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.008s
I20240503 14:13:33.401430 20383 raft_consensus.cc:348] T c45ceab3a36d4225aed74654ed530c9d P 64ad8799c8df4b7589869baffb93b68d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.401803 20383 raft_consensus.cc:374] T c45ceab3a36d4225aed74654ed530c9d P 64ad8799c8df4b7589869baffb93b68d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:33.401922 20383 raft_consensus.cc:729] T c45ceab3a36d4225aed74654ed530c9d P 64ad8799c8df4b7589869baffb93b68d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64ad8799c8df4b7589869baffb93b68d, State: Initialized, Role: FOLLOWER
I20240503 14:13:33.401963 20380 raft_consensus.cc:348] T ad711d78ee524d979867b65906abe456 P a5d66df07715448fab64928f5bc8299a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.402216 20380 raft_consensus.cc:374] T ad711d78ee524d979867b65906abe456 P a5d66df07715448fab64928f5bc8299a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:33.402159 20383 consensus_queue.cc:260] T c45ceab3a36d4225aed74654ed530c9d P 64ad8799c8df4b7589869baffb93b68d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.402316 20380 raft_consensus.cc:729] T ad711d78ee524d979867b65906abe456 P a5d66df07715448fab64928f5bc8299a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5d66df07715448fab64928f5bc8299a, State: Initialized, Role: FOLLOWER
I20240503 14:13:33.402323 20384 raft_consensus.cc:348] T 7905fd813a2d4a8aa8b2ac1e752b37dc P fceaf96557de46c3986c306fa4ac3b9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.402663 20384 raft_consensus.cc:374] T 7905fd813a2d4a8aa8b2ac1e752b37dc P fceaf96557de46c3986c306fa4ac3b9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:33.402635 20380 consensus_queue.cc:260] T ad711d78ee524d979867b65906abe456 P a5d66df07715448fab64928f5bc8299a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.402804 20384 raft_consensus.cc:729] T 7905fd813a2d4a8aa8b2ac1e752b37dc P fceaf96557de46c3986c306fa4ac3b9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fceaf96557de46c3986c306fa4ac3b9e, State: Initialized, Role: FOLLOWER
I20240503 14:13:33.402830 20383 ts_tablet_manager.cc:1398] T c45ceab3a36d4225aed74654ed530c9d P 64ad8799c8df4b7589869baffb93b68d: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 14:13:33.403230 20380 ts_tablet_manager.cc:1398] T ad711d78ee524d979867b65906abe456 P a5d66df07715448fab64928f5bc8299a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:33.403270 20383 tablet_bootstrap.cc:492] T 7905fd813a2d4a8aa8b2ac1e752b37dc P 64ad8799c8df4b7589869baffb93b68d: Bootstrap starting.
I20240503 14:13:33.403234 20384 consensus_queue.cc:260] T 7905fd813a2d4a8aa8b2ac1e752b37dc P fceaf96557de46c3986c306fa4ac3b9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.403856 20384 ts_tablet_manager.cc:1398] T 7905fd813a2d4a8aa8b2ac1e752b37dc P fceaf96557de46c3986c306fa4ac3b9e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:33.405207 20383 tablet_bootstrap.cc:654] T 7905fd813a2d4a8aa8b2ac1e752b37dc P 64ad8799c8df4b7589869baffb93b68d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:33.408077 20383 tablet_bootstrap.cc:492] T 7905fd813a2d4a8aa8b2ac1e752b37dc P 64ad8799c8df4b7589869baffb93b68d: No bootstrap required, opened a new log
I20240503 14:13:33.408228 20383 ts_tablet_manager.cc:1367] T 7905fd813a2d4a8aa8b2ac1e752b37dc P 64ad8799c8df4b7589869baffb93b68d: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240503 14:13:33.408738 20383 raft_consensus.cc:348] T 7905fd813a2d4a8aa8b2ac1e752b37dc P 64ad8799c8df4b7589869baffb93b68d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.408902 20383 raft_consensus.cc:374] T 7905fd813a2d4a8aa8b2ac1e752b37dc P 64ad8799c8df4b7589869baffb93b68d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:33.408979 20383 raft_consensus.cc:729] T 7905fd813a2d4a8aa8b2ac1e752b37dc P 64ad8799c8df4b7589869baffb93b68d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64ad8799c8df4b7589869baffb93b68d, State: Initialized, Role: FOLLOWER
I20240503 14:13:33.409204 20383 consensus_queue.cc:260] T 7905fd813a2d4a8aa8b2ac1e752b37dc P 64ad8799c8df4b7589869baffb93b68d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.409596 20383 ts_tablet_manager.cc:1398] T 7905fd813a2d4a8aa8b2ac1e752b37dc P 64ad8799c8df4b7589869baffb93b68d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:13:33.413214 20437 raft_consensus.cc:482] T 7a82ae1f4c66455caea352a304af2b51 P a5d66df07715448fab64928f5bc8299a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:33.413355 20437 raft_consensus.cc:504] T 7a82ae1f4c66455caea352a304af2b51 P a5d66df07715448fab64928f5bc8299a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.413904 20437 leader_election.cc:288] T 7a82ae1f4c66455caea352a304af2b51 P a5d66df07715448fab64928f5bc8299a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64ad8799c8df4b7589869baffb93b68d (127.16.94.129:37429), fceaf96557de46c3986c306fa4ac3b9e (127.16.94.131:39581)
I20240503 14:13:33.414278 20335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a82ae1f4c66455caea352a304af2b51" candidate_uuid: "a5d66df07715448fab64928f5bc8299a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fceaf96557de46c3986c306fa4ac3b9e" is_pre_election: true
I20240503 14:13:33.414239 20184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a82ae1f4c66455caea352a304af2b51" candidate_uuid: "a5d66df07715448fab64928f5bc8299a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64ad8799c8df4b7589869baffb93b68d" is_pre_election: true
I20240503 14:13:33.414492 20335 raft_consensus.cc:2454] T 7a82ae1f4c66455caea352a304af2b51 P fceaf96557de46c3986c306fa4ac3b9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5d66df07715448fab64928f5bc8299a in term 0.
I20240503 14:13:33.414492 20184 raft_consensus.cc:2454] T 7a82ae1f4c66455caea352a304af2b51 P 64ad8799c8df4b7589869baffb93b68d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5d66df07715448fab64928f5bc8299a in term 0.
I20240503 14:13:33.414850 20226 leader_election.cc:302] T 7a82ae1f4c66455caea352a304af2b51 P a5d66df07715448fab64928f5bc8299a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5d66df07715448fab64928f5bc8299a, fceaf96557de46c3986c306fa4ac3b9e; no voters:
I20240503 14:13:33.415120 20437 raft_consensus.cc:2781] T 7a82ae1f4c66455caea352a304af2b51 P a5d66df07715448fab64928f5bc8299a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:33.415199 20437 raft_consensus.cc:482] T 7a82ae1f4c66455caea352a304af2b51 P a5d66df07715448fab64928f5bc8299a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:33.415271 20437 raft_consensus.cc:3037] T 7a82ae1f4c66455caea352a304af2b51 P a5d66df07715448fab64928f5bc8299a [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:33.417047 20437 raft_consensus.cc:504] T 7a82ae1f4c66455caea352a304af2b51 P a5d66df07715448fab64928f5bc8299a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.417563 20437 leader_election.cc:288] T 7a82ae1f4c66455caea352a304af2b51 P a5d66df07715448fab64928f5bc8299a [CANDIDATE]: Term 1 election: Requested vote from peers 64ad8799c8df4b7589869baffb93b68d (127.16.94.129:37429), fceaf96557de46c3986c306fa4ac3b9e (127.16.94.131:39581)
I20240503 14:13:33.417999 20184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a82ae1f4c66455caea352a304af2b51" candidate_uuid: "a5d66df07715448fab64928f5bc8299a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64ad8799c8df4b7589869baffb93b68d"
I20240503 14:13:33.418004 20335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a82ae1f4c66455caea352a304af2b51" candidate_uuid: "a5d66df07715448fab64928f5bc8299a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fceaf96557de46c3986c306fa4ac3b9e"
I20240503 14:13:33.418174 20184 raft_consensus.cc:3037] T 7a82ae1f4c66455caea352a304af2b51 P 64ad8799c8df4b7589869baffb93b68d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:33.418224 20335 raft_consensus.cc:3037] T 7a82ae1f4c66455caea352a304af2b51 P fceaf96557de46c3986c306fa4ac3b9e [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:33.420223 20184 raft_consensus.cc:2454] T 7a82ae1f4c66455caea352a304af2b51 P 64ad8799c8df4b7589869baffb93b68d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5d66df07715448fab64928f5bc8299a in term 1.
I20240503 14:13:33.420462 20335 raft_consensus.cc:2454] T 7a82ae1f4c66455caea352a304af2b51 P fceaf96557de46c3986c306fa4ac3b9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5d66df07715448fab64928f5bc8299a in term 1.
I20240503 14:13:33.420588 20224 leader_election.cc:302] T 7a82ae1f4c66455caea352a304af2b51 P a5d66df07715448fab64928f5bc8299a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64ad8799c8df4b7589869baffb93b68d, a5d66df07715448fab64928f5bc8299a; no voters:
I20240503 14:13:33.420893 20437 raft_consensus.cc:2781] T 7a82ae1f4c66455caea352a304af2b51 P a5d66df07715448fab64928f5bc8299a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:33.421036 20437 raft_consensus.cc:686] T 7a82ae1f4c66455caea352a304af2b51 P a5d66df07715448fab64928f5bc8299a [term 1 LEADER]: Becoming Leader. State: Replica: a5d66df07715448fab64928f5bc8299a, State: Running, Role: LEADER
I20240503 14:13:33.421241 20437 consensus_queue.cc:237] T 7a82ae1f4c66455caea352a304af2b51 P a5d66df07715448fab64928f5bc8299a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.423784 20081 catalog_manager.cc:5484] T 7a82ae1f4c66455caea352a304af2b51 P a5d66df07715448fab64928f5bc8299a reported cstate change: term changed from 0 to 1, leader changed from <none> to a5d66df07715448fab64928f5bc8299a (127.16.94.130). New cstate: current_term: 1 leader_uuid: "a5d66df07715448fab64928f5bc8299a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:33.438066 20437 raft_consensus.cc:482] T 5484fffa7f6b4e64ba4798e8816b6c61 P a5d66df07715448fab64928f5bc8299a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:33.438259 20437 raft_consensus.cc:504] T 5484fffa7f6b4e64ba4798e8816b6c61 P a5d66df07715448fab64928f5bc8299a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.438915 20437 leader_election.cc:288] T 5484fffa7f6b4e64ba4798e8816b6c61 P a5d66df07715448fab64928f5bc8299a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64ad8799c8df4b7589869baffb93b68d (127.16.94.129:37429), fceaf96557de46c3986c306fa4ac3b9e (127.16.94.131:39581)
I20240503 14:13:33.439368 20184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5484fffa7f6b4e64ba4798e8816b6c61" candidate_uuid: "a5d66df07715448fab64928f5bc8299a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64ad8799c8df4b7589869baffb93b68d" is_pre_election: true
I20240503 14:13:33.439416 20335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5484fffa7f6b4e64ba4798e8816b6c61" candidate_uuid: "a5d66df07715448fab64928f5bc8299a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fceaf96557de46c3986c306fa4ac3b9e" is_pre_election: true
I20240503 14:13:33.439606 20184 raft_consensus.cc:2454] T 5484fffa7f6b4e64ba4798e8816b6c61 P 64ad8799c8df4b7589869baffb93b68d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5d66df07715448fab64928f5bc8299a in term 0.
I20240503 14:13:33.439646 20335 raft_consensus.cc:2454] T 5484fffa7f6b4e64ba4798e8816b6c61 P fceaf96557de46c3986c306fa4ac3b9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5d66df07715448fab64928f5bc8299a in term 0.
I20240503 14:13:33.440016 20224 leader_election.cc:302] T 5484fffa7f6b4e64ba4798e8816b6c61 P a5d66df07715448fab64928f5bc8299a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64ad8799c8df4b7589869baffb93b68d, a5d66df07715448fab64928f5bc8299a; no voters:
I20240503 14:13:33.440289 20437 raft_consensus.cc:2781] T 5484fffa7f6b4e64ba4798e8816b6c61 P a5d66df07715448fab64928f5bc8299a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:33.440380 20437 raft_consensus.cc:482] T 5484fffa7f6b4e64ba4798e8816b6c61 P a5d66df07715448fab64928f5bc8299a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:33.440444 20437 raft_consensus.cc:3037] T 5484fffa7f6b4e64ba4798e8816b6c61 P a5d66df07715448fab64928f5bc8299a [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:33.442337 20437 raft_consensus.cc:504] T 5484fffa7f6b4e64ba4798e8816b6c61 P a5d66df07715448fab64928f5bc8299a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.442927 20437 leader_election.cc:288] T 5484fffa7f6b4e64ba4798e8816b6c61 P a5d66df07715448fab64928f5bc8299a [CANDIDATE]: Term 1 election: Requested vote from peers 64ad8799c8df4b7589869baffb93b68d (127.16.94.129:37429), fceaf96557de46c3986c306fa4ac3b9e (127.16.94.131:39581)
I20240503 14:13:33.443285 20184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5484fffa7f6b4e64ba4798e8816b6c61" candidate_uuid: "a5d66df07715448fab64928f5bc8299a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64ad8799c8df4b7589869baffb93b68d"
I20240503 14:13:33.443393 20335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5484fffa7f6b4e64ba4798e8816b6c61" candidate_uuid: "a5d66df07715448fab64928f5bc8299a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fceaf96557de46c3986c306fa4ac3b9e"
I20240503 14:13:33.443466 20184 raft_consensus.cc:3037] T 5484fffa7f6b4e64ba4798e8816b6c61 P 64ad8799c8df4b7589869baffb93b68d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:33.443569 20335 raft_consensus.cc:3037] T 5484fffa7f6b4e64ba4798e8816b6c61 P fceaf96557de46c3986c306fa4ac3b9e [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:33.445542 20184 raft_consensus.cc:2454] T 5484fffa7f6b4e64ba4798e8816b6c61 P 64ad8799c8df4b7589869baffb93b68d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5d66df07715448fab64928f5bc8299a in term 1.
I20240503 14:13:33.445727 20335 raft_consensus.cc:2454] T 5484fffa7f6b4e64ba4798e8816b6c61 P fceaf96557de46c3986c306fa4ac3b9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5d66df07715448fab64928f5bc8299a in term 1.
I20240503 14:13:33.445960 20224 leader_election.cc:302] T 5484fffa7f6b4e64ba4798e8816b6c61 P a5d66df07715448fab64928f5bc8299a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64ad8799c8df4b7589869baffb93b68d, a5d66df07715448fab64928f5bc8299a; no voters:
I20240503 14:13:33.446234 20437 raft_consensus.cc:2781] T 5484fffa7f6b4e64ba4798e8816b6c61 P a5d66df07715448fab64928f5bc8299a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:33.446388 20437 raft_consensus.cc:686] T 5484fffa7f6b4e64ba4798e8816b6c61 P a5d66df07715448fab64928f5bc8299a [term 1 LEADER]: Becoming Leader. State: Replica: a5d66df07715448fab64928f5bc8299a, State: Running, Role: LEADER
I20240503 14:13:33.446666 20437 consensus_queue.cc:237] T 5484fffa7f6b4e64ba4798e8816b6c61 P a5d66df07715448fab64928f5bc8299a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.448990 20081 catalog_manager.cc:5484] T 5484fffa7f6b4e64ba4798e8816b6c61 P a5d66df07715448fab64928f5bc8299a reported cstate change: term changed from 0 to 1, leader changed from <none> to a5d66df07715448fab64928f5bc8299a (127.16.94.130). New cstate: current_term: 1 leader_uuid: "a5d66df07715448fab64928f5bc8299a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:33.464023 20437 raft_consensus.cc:482] T 0eb7552695304a1ea6f8e02f189f7295 P a5d66df07715448fab64928f5bc8299a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:33.464231 20437 raft_consensus.cc:504] T 0eb7552695304a1ea6f8e02f189f7295 P a5d66df07715448fab64928f5bc8299a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.464849 20437 leader_election.cc:288] T 0eb7552695304a1ea6f8e02f189f7295 P a5d66df07715448fab64928f5bc8299a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64ad8799c8df4b7589869baffb93b68d (127.16.94.129:37429), fceaf96557de46c3986c306fa4ac3b9e (127.16.94.131:39581)
I20240503 14:13:33.465328 20335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eb7552695304a1ea6f8e02f189f7295" candidate_uuid: "a5d66df07715448fab64928f5bc8299a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fceaf96557de46c3986c306fa4ac3b9e" is_pre_election: true
I20240503 14:13:33.465337 20184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eb7552695304a1ea6f8e02f189f7295" candidate_uuid: "a5d66df07715448fab64928f5bc8299a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64ad8799c8df4b7589869baffb93b68d" is_pre_election: true
I20240503 14:13:33.465629 20184 raft_consensus.cc:2379] T 0eb7552695304a1ea6f8e02f189f7295 P 64ad8799c8df4b7589869baffb93b68d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a5d66df07715448fab64928f5bc8299a in current term 1: Already voted for candidate fceaf96557de46c3986c306fa4ac3b9e in this term.
I20240503 14:13:33.466054 20224 leader_election.cc:302] T 0eb7552695304a1ea6f8e02f189f7295 P a5d66df07715448fab64928f5bc8299a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5d66df07715448fab64928f5bc8299a; no voters: 64ad8799c8df4b7589869baffb93b68d, fceaf96557de46c3986c306fa4ac3b9e
I20240503 14:13:33.466315 20437 raft_consensus.cc:3037] T 0eb7552695304a1ea6f8e02f189f7295 P a5d66df07715448fab64928f5bc8299a [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:33.468114 20437 raft_consensus.cc:2726] T 0eb7552695304a1ea6f8e02f189f7295 P a5d66df07715448fab64928f5bc8299a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:13:33.484836 20437 raft_consensus.cc:482] T 7905fd813a2d4a8aa8b2ac1e752b37dc P a5d66df07715448fab64928f5bc8299a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:33.485003 20437 raft_consensus.cc:504] T 7905fd813a2d4a8aa8b2ac1e752b37dc P a5d66df07715448fab64928f5bc8299a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.485555 20437 leader_election.cc:288] T 7905fd813a2d4a8aa8b2ac1e752b37dc P a5d66df07715448fab64928f5bc8299a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64ad8799c8df4b7589869baffb93b68d (127.16.94.129:37429), fceaf96557de46c3986c306fa4ac3b9e (127.16.94.131:39581)
I20240503 14:13:33.486086 20335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7905fd813a2d4a8aa8b2ac1e752b37dc" candidate_uuid: "a5d66df07715448fab64928f5bc8299a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fceaf96557de46c3986c306fa4ac3b9e" is_pre_election: true
I20240503 14:13:33.486135 20184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7905fd813a2d4a8aa8b2ac1e752b37dc" candidate_uuid: "a5d66df07715448fab64928f5bc8299a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64ad8799c8df4b7589869baffb93b68d" is_pre_election: true
I20240503 14:13:33.486277 20335 raft_consensus.cc:2454] T 7905fd813a2d4a8aa8b2ac1e752b37dc P fceaf96557de46c3986c306fa4ac3b9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5d66df07715448fab64928f5bc8299a in term 0.
I20240503 14:13:33.486340 20184 raft_consensus.cc:2454] T 7905fd813a2d4a8aa8b2ac1e752b37dc P 64ad8799c8df4b7589869baffb93b68d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5d66df07715448fab64928f5bc8299a in term 0.
I20240503 14:13:33.486652 20226 leader_election.cc:302] T 7905fd813a2d4a8aa8b2ac1e752b37dc P a5d66df07715448fab64928f5bc8299a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5d66df07715448fab64928f5bc8299a, fceaf96557de46c3986c306fa4ac3b9e; no voters:
I20240503 14:13:33.486903 20437 raft_consensus.cc:2781] T 7905fd813a2d4a8aa8b2ac1e752b37dc P a5d66df07715448fab64928f5bc8299a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:33.487005 20437 raft_consensus.cc:482] T 7905fd813a2d4a8aa8b2ac1e752b37dc P a5d66df07715448fab64928f5bc8299a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:33.487095 20437 raft_consensus.cc:3037] T 7905fd813a2d4a8aa8b2ac1e752b37dc P a5d66df07715448fab64928f5bc8299a [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:33.488955 20437 raft_consensus.cc:504] T 7905fd813a2d4a8aa8b2ac1e752b37dc P a5d66df07715448fab64928f5bc8299a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.489439 20437 leader_election.cc:288] T 7905fd813a2d4a8aa8b2ac1e752b37dc P a5d66df07715448fab64928f5bc8299a [CANDIDATE]: Term 1 election: Requested vote from peers 64ad8799c8df4b7589869baffb93b68d (127.16.94.129:37429), fceaf96557de46c3986c306fa4ac3b9e (127.16.94.131:39581)
I20240503 14:13:33.489933 20335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7905fd813a2d4a8aa8b2ac1e752b37dc" candidate_uuid: "a5d66df07715448fab64928f5bc8299a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fceaf96557de46c3986c306fa4ac3b9e"
I20240503 14:13:33.489981 20184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7905fd813a2d4a8aa8b2ac1e752b37dc" candidate_uuid: "a5d66df07715448fab64928f5bc8299a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64ad8799c8df4b7589869baffb93b68d"
I20240503 14:13:33.490098 20335 raft_consensus.cc:3037] T 7905fd813a2d4a8aa8b2ac1e752b37dc P fceaf96557de46c3986c306fa4ac3b9e [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:33.490181 20184 raft_consensus.cc:3037] T 7905fd813a2d4a8aa8b2ac1e752b37dc P 64ad8799c8df4b7589869baffb93b68d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:33.492038 20184 raft_consensus.cc:2454] T 7905fd813a2d4a8aa8b2ac1e752b37dc P 64ad8799c8df4b7589869baffb93b68d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5d66df07715448fab64928f5bc8299a in term 1.
I20240503 14:13:33.492061 20335 raft_consensus.cc:2454] T 7905fd813a2d4a8aa8b2ac1e752b37dc P fceaf96557de46c3986c306fa4ac3b9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5d66df07715448fab64928f5bc8299a in term 1.
I20240503 14:13:33.492385 20226 leader_election.cc:302] T 7905fd813a2d4a8aa8b2ac1e752b37dc P a5d66df07715448fab64928f5bc8299a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5d66df07715448fab64928f5bc8299a, fceaf96557de46c3986c306fa4ac3b9e; no voters:
I20240503 14:13:33.492638 20437 raft_consensus.cc:2781] T 7905fd813a2d4a8aa8b2ac1e752b37dc P a5d66df07715448fab64928f5bc8299a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:33.492766 20437 raft_consensus.cc:686] T 7905fd813a2d4a8aa8b2ac1e752b37dc P a5d66df07715448fab64928f5bc8299a [term 1 LEADER]: Becoming Leader. State: Replica: a5d66df07715448fab64928f5bc8299a, State: Running, Role: LEADER
I20240503 14:13:33.493019 20437 consensus_queue.cc:237] T 7905fd813a2d4a8aa8b2ac1e752b37dc P a5d66df07715448fab64928f5bc8299a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.495537 20081 catalog_manager.cc:5484] T 7905fd813a2d4a8aa8b2ac1e752b37dc P a5d66df07715448fab64928f5bc8299a reported cstate change: term changed from 0 to 1, leader changed from <none> to a5d66df07715448fab64928f5bc8299a (127.16.94.130). New cstate: current_term: 1 leader_uuid: "a5d66df07715448fab64928f5bc8299a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:33.506865 20388 raft_consensus.cc:482] T d4743d5a73c54086b277f4c4b1a248fd P fceaf96557de46c3986c306fa4ac3b9e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:33.507047 20388 raft_consensus.cc:504] T d4743d5a73c54086b277f4c4b1a248fd P fceaf96557de46c3986c306fa4ac3b9e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.507628 20388 leader_election.cc:288] T d4743d5a73c54086b277f4c4b1a248fd P fceaf96557de46c3986c306fa4ac3b9e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64ad8799c8df4b7589869baffb93b68d (127.16.94.129:37429), a5d66df07715448fab64928f5bc8299a (127.16.94.130:39295)
I20240503 14:13:33.508015 20184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4743d5a73c54086b277f4c4b1a248fd" candidate_uuid: "fceaf96557de46c3986c306fa4ac3b9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64ad8799c8df4b7589869baffb93b68d" is_pre_election: true
I20240503 14:13:33.508065 20261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4743d5a73c54086b277f4c4b1a248fd" candidate_uuid: "fceaf96557de46c3986c306fa4ac3b9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5d66df07715448fab64928f5bc8299a" is_pre_election: true
I20240503 14:13:33.508299 20184 raft_consensus.cc:2379] T d4743d5a73c54086b277f4c4b1a248fd P 64ad8799c8df4b7589869baffb93b68d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fceaf96557de46c3986c306fa4ac3b9e in current term 1: Already voted for candidate a5d66df07715448fab64928f5bc8299a in this term.
I20240503 14:13:33.508664 20298 leader_election.cc:302] T d4743d5a73c54086b277f4c4b1a248fd P fceaf96557de46c3986c306fa4ac3b9e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fceaf96557de46c3986c306fa4ac3b9e; no voters: 64ad8799c8df4b7589869baffb93b68d, a5d66df07715448fab64928f5bc8299a
I20240503 14:13:33.508898 20388 raft_consensus.cc:3037] T d4743d5a73c54086b277f4c4b1a248fd P fceaf96557de46c3986c306fa4ac3b9e [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:33.510748 20388 raft_consensus.cc:2726] T d4743d5a73c54086b277f4c4b1a248fd P fceaf96557de46c3986c306fa4ac3b9e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:13:33.587268 20437 raft_consensus.cc:482] T ad711d78ee524d979867b65906abe456 P a5d66df07715448fab64928f5bc8299a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:33.587486 20437 raft_consensus.cc:504] T ad711d78ee524d979867b65906abe456 P a5d66df07715448fab64928f5bc8299a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.588229 20437 leader_election.cc:288] T ad711d78ee524d979867b65906abe456 P a5d66df07715448fab64928f5bc8299a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64ad8799c8df4b7589869baffb93b68d (127.16.94.129:37429), fceaf96557de46c3986c306fa4ac3b9e (127.16.94.131:39581)
I20240503 14:13:33.588507 20184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad711d78ee524d979867b65906abe456" candidate_uuid: "a5d66df07715448fab64928f5bc8299a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64ad8799c8df4b7589869baffb93b68d" is_pre_election: true
I20240503 14:13:33.588819 20184 raft_consensus.cc:2379] T ad711d78ee524d979867b65906abe456 P 64ad8799c8df4b7589869baffb93b68d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a5d66df07715448fab64928f5bc8299a in current term 1: Already voted for candidate fceaf96557de46c3986c306fa4ac3b9e in this term.
I20240503 14:13:33.588864 20335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad711d78ee524d979867b65906abe456" candidate_uuid: "a5d66df07715448fab64928f5bc8299a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fceaf96557de46c3986c306fa4ac3b9e" is_pre_election: true
I20240503 14:13:33.589299 20226 leader_election.cc:302] T ad711d78ee524d979867b65906abe456 P a5d66df07715448fab64928f5bc8299a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5d66df07715448fab64928f5bc8299a; no voters: 64ad8799c8df4b7589869baffb93b68d, fceaf96557de46c3986c306fa4ac3b9e
I20240503 14:13:33.589610 20437 raft_consensus.cc:3037] T ad711d78ee524d979867b65906abe456 P a5d66df07715448fab64928f5bc8299a [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:33.591625 20437 raft_consensus.cc:2726] T ad711d78ee524d979867b65906abe456 P a5d66df07715448fab64928f5bc8299a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:13:33.621119 20335 raft_consensus.cc:1261] T 29f4ee03917646f6a5374217bed7a1be P fceaf96557de46c3986c306fa4ac3b9e [term 1 FOLLOWER]: Refusing update from remote peer 64ad8799c8df4b7589869baffb93b68d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:33.621198 20261 raft_consensus.cc:1261] T 29f4ee03917646f6a5374217bed7a1be P a5d66df07715448fab64928f5bc8299a [term 1 FOLLOWER]: Refusing update from remote peer 64ad8799c8df4b7589869baffb93b68d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:33.622649 20421 consensus_queue.cc:1035] T 29f4ee03917646f6a5374217bed7a1be P 64ad8799c8df4b7589869baffb93b68d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:33.624285 20389 consensus_queue.cc:1035] T 29f4ee03917646f6a5374217bed7a1be P 64ad8799c8df4b7589869baffb93b68d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:13:33.626011 20261 raft_consensus.cc:1261] T 220ef8bab4694ace9bdf0fce06226efa P a5d66df07715448fab64928f5bc8299a [term 1 FOLLOWER]: Refusing update from remote peer 64ad8799c8df4b7589869baffb93b68d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:33.628543 20335 raft_consensus.cc:3037] T 220ef8bab4694ace9bdf0fce06226efa P fceaf96557de46c3986c306fa4ac3b9e [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:33.629344 20261 raft_consensus.cc:1261] T ad711d78ee524d979867b65906abe456 P a5d66df07715448fab64928f5bc8299a [term 1 FOLLOWER]: Refusing update from remote peer fceaf96557de46c3986c306fa4ac3b9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:33.629462 20259 raft_consensus.cc:1261] T 0eb7552695304a1ea6f8e02f189f7295 P a5d66df07715448fab64928f5bc8299a [term 1 FOLLOWER]: Refusing update from remote peer fceaf96557de46c3986c306fa4ac3b9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:33.630281 20184 raft_consensus.cc:1261] T 0eb7552695304a1ea6f8e02f189f7295 P 64ad8799c8df4b7589869baffb93b68d [term 1 FOLLOWER]: Refusing update from remote peer fceaf96557de46c3986c306fa4ac3b9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:33.631786 20420 consensus_queue.cc:1035] T 0eb7552695304a1ea6f8e02f189f7295 P fceaf96557de46c3986c306fa4ac3b9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:13:33.630929 20389 consensus_queue.cc:1035] T 220ef8bab4694ace9bdf0fce06226efa P 64ad8799c8df4b7589869baffb93b68d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20240503 14:13:33.633118 20185 raft_consensus.cc:1261] T ad711d78ee524d979867b65906abe456 P 64ad8799c8df4b7589869baffb93b68d [term 1 FOLLOWER]: Refusing update from remote peer fceaf96557de46c3986c306fa4ac3b9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:33.634004 20420 consensus_queue.cc:1035] T 0eb7552695304a1ea6f8e02f189f7295 P fceaf96557de46c3986c306fa4ac3b9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:13:33.634940 20388 consensus_queue.cc:1035] T ad711d78ee524d979867b65906abe456 P fceaf96557de46c3986c306fa4ac3b9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20240503 14:13:33.639305 20431 consensus_queue.cc:1035] T ad711d78ee524d979867b65906abe456 P fceaf96557de46c3986c306fa4ac3b9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
I20240503 14:13:33.648483 20136 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:13:33.651034 20335 raft_consensus.cc:1261] T 220ef8bab4694ace9bdf0fce06226efa P fceaf96557de46c3986c306fa4ac3b9e [term 1 FOLLOWER]: Refusing update from remote peer 64ad8799c8df4b7589869baffb93b68d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:33.652475 20389 consensus_queue.cc:1035] T 220ef8bab4694ace9bdf0fce06226efa P 64ad8799c8df4b7589869baffb93b68d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:13:33.678701 20333 raft_consensus.cc:1261] T 3054a11a8da4470ea5241b92d164d6a8 P fceaf96557de46c3986c306fa4ac3b9e [term 1 FOLLOWER]: Refusing update from remote peer a5d66df07715448fab64928f5bc8299a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:33.693046 20332 raft_consensus.cc:1261] T 7905fd813a2d4a8aa8b2ac1e752b37dc P fceaf96557de46c3986c306fa4ac3b9e [term 1 FOLLOWER]: Refusing update from remote peer a5d66df07715448fab64928f5bc8299a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:33.717072 20437 consensus_queue.cc:1035] T 3054a11a8da4470ea5241b92d164d6a8 P a5d66df07715448fab64928f5bc8299a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.038s
I20240503 14:13:33.722136 20333 raft_consensus.cc:1261] T c45ceab3a36d4225aed74654ed530c9d P fceaf96557de46c3986c306fa4ac3b9e [term 1 FOLLOWER]: Refusing update from remote peer a5d66df07715448fab64928f5bc8299a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:33.730047 20331 raft_consensus.cc:3037] T 44a4d2b820fc4d0da06c659fe98fe57c P fceaf96557de46c3986c306fa4ac3b9e [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:33.730161 20332 raft_consensus.cc:1261] T d4743d5a73c54086b277f4c4b1a248fd P fceaf96557de46c3986c306fa4ac3b9e [term 1 FOLLOWER]: Refusing update from remote peer a5d66df07715448fab64928f5bc8299a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:33.732192 20382 consensus_queue.cc:1035] T 7905fd813a2d4a8aa8b2ac1e752b37dc P a5d66df07715448fab64928f5bc8299a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:33.788025 20331 raft_consensus.cc:1261] T 44a4d2b820fc4d0da06c659fe98fe57c P fceaf96557de46c3986c306fa4ac3b9e [term 1 FOLLOWER]: Refusing update from remote peer a5d66df07715448fab64928f5bc8299a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:33.798328 20382 consensus_queue.cc:1035] T d4743d5a73c54086b277f4c4b1a248fd P a5d66df07715448fab64928f5bc8299a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.066s
I20240503 14:13:33.803988 20177 raft_consensus.cc:3037] T 904450ab5e8b4250b7af18b98d2b0985 P 64ad8799c8df4b7589869baffb93b68d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:33.804286 20182 raft_consensus.cc:1261] T 44a4d2b820fc4d0da06c659fe98fe57c P 64ad8799c8df4b7589869baffb93b68d [term 1 FOLLOWER]: Refusing update from remote peer a5d66df07715448fab64928f5bc8299a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:33.804549 20180 raft_consensus.cc:1261] T d4743d5a73c54086b277f4c4b1a248fd P 64ad8799c8df4b7589869baffb93b68d [term 1 FOLLOWER]: Refusing update from remote peer a5d66df07715448fab64928f5bc8299a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:33.804826 20178 raft_consensus.cc:1261] T 5484fffa7f6b4e64ba4798e8816b6c61 P 64ad8799c8df4b7589869baffb93b68d [term 1 FOLLOWER]: Refusing update from remote peer a5d66df07715448fab64928f5bc8299a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:33.804926 20181 raft_consensus.cc:1261] T 7905fd813a2d4a8aa8b2ac1e752b37dc P 64ad8799c8df4b7589869baffb93b68d [term 1 FOLLOWER]: Refusing update from remote peer a5d66df07715448fab64928f5bc8299a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:33.805164 20431 raft_consensus.cc:482] T 44a4d2b820fc4d0da06c659fe98fe57c P fceaf96557de46c3986c306fa4ac3b9e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:33.808534 20179 raft_consensus.cc:3037] T c45ceab3a36d4225aed74654ed530c9d P 64ad8799c8df4b7589869baffb93b68d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:33.808920 20183 raft_consensus.cc:1261] T 3054a11a8da4470ea5241b92d164d6a8 P 64ad8799c8df4b7589869baffb93b68d [term 1 FOLLOWER]: Refusing update from remote peer a5d66df07715448fab64928f5bc8299a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:33.814287 20431 raft_consensus.cc:504] T 44a4d2b820fc4d0da06c659fe98fe57c P fceaf96557de46c3986c306fa4ac3b9e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
W20240503 14:13:33.816884 20446 log.cc:924] Time spent T 5484fffa7f6b4e64ba4798e8816b6c61 P a5d66df07715448fab64928f5bc8299a: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
I20240503 14:13:33.822314 20479 consensus_queue.cc:1035] T 44a4d2b820fc4d0da06c659fe98fe57c P a5d66df07715448fab64928f5bc8299a [LEADER]: Connected to new peer: Peer: permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:33.822626 20431 leader_election.cc:288] T 44a4d2b820fc4d0da06c659fe98fe57c P fceaf96557de46c3986c306fa4ac3b9e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 64ad8799c8df4b7589869baffb93b68d (127.16.94.129:37429), a5d66df07715448fab64928f5bc8299a (127.16.94.130:39295)
I20240503 14:13:33.822973 20479 consensus_queue.cc:1035] T d4743d5a73c54086b277f4c4b1a248fd P a5d66df07715448fab64928f5bc8299a [LEADER]: Connected to new peer: Peer: permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:33.823293 20178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44a4d2b820fc4d0da06c659fe98fe57c" candidate_uuid: "fceaf96557de46c3986c306fa4ac3b9e" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64ad8799c8df4b7589869baffb93b68d" is_pre_election: true
I20240503 14:13:33.823565 20178 raft_consensus.cc:2454] T 44a4d2b820fc4d0da06c659fe98fe57c P 64ad8799c8df4b7589869baffb93b68d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fceaf96557de46c3986c306fa4ac3b9e in term 1.
I20240503 14:13:33.824026 20298 leader_election.cc:302] T 44a4d2b820fc4d0da06c659fe98fe57c P fceaf96557de46c3986c306fa4ac3b9e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64ad8799c8df4b7589869baffb93b68d, fceaf96557de46c3986c306fa4ac3b9e; no voters:
I20240503 14:13:33.823951 20257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44a4d2b820fc4d0da06c659fe98fe57c" candidate_uuid: "fceaf96557de46c3986c306fa4ac3b9e" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5d66df07715448fab64928f5bc8299a" is_pre_election: true
I20240503 14:13:33.824939 20431 raft_consensus.cc:2781] T 44a4d2b820fc4d0da06c659fe98fe57c P fceaf96557de46c3986c306fa4ac3b9e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:13:33.825044 20431 raft_consensus.cc:482] T 44a4d2b820fc4d0da06c659fe98fe57c P fceaf96557de46c3986c306fa4ac3b9e [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:33.825160 20431 raft_consensus.cc:3037] T 44a4d2b820fc4d0da06c659fe98fe57c P fceaf96557de46c3986c306fa4ac3b9e [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:33.834561 20437 consensus_queue.cc:1035] T 5484fffa7f6b4e64ba4798e8816b6c61 P a5d66df07715448fab64928f5bc8299a [LEADER]: Connected to new peer: Peer: permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20240503 14:13:33.835207 20382 consensus_queue.cc:1035] T 3054a11a8da4470ea5241b92d164d6a8 P a5d66df07715448fab64928f5bc8299a [LEADER]: Connected to new peer: Peer: permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20240503 14:13:33.835692 20408 consensus_queue.cc:1035] T 7905fd813a2d4a8aa8b2ac1e752b37dc P a5d66df07715448fab64928f5bc8299a [LEADER]: Connected to new peer: Peer: permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
W20240503 14:13:33.836932 20480 log.cc:924] Time spent T 3054a11a8da4470ea5241b92d164d6a8 P fceaf96557de46c3986c306fa4ac3b9e: Append to log took a long time: real 0.072s user 0.000s sys 0.000s
I20240503 14:13:33.845759 20431 raft_consensus.cc:504] T 44a4d2b820fc4d0da06c659fe98fe57c P fceaf96557de46c3986c306fa4ac3b9e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
W20240503 14:13:33.834368 20226 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.94.131:39581, user_credentials={real_user=slave}} blocked reactor thread for 103347us
I20240503 14:13:33.852869 20431 leader_election.cc:288] T 44a4d2b820fc4d0da06c659fe98fe57c P fceaf96557de46c3986c306fa4ac3b9e [CANDIDATE]: Term 2 election: Requested vote from peers 64ad8799c8df4b7589869baffb93b68d (127.16.94.129:37429), a5d66df07715448fab64928f5bc8299a (127.16.94.130:39295)
I20240503 14:13:33.862517 20179 raft_consensus.cc:1261] T c45ceab3a36d4225aed74654ed530c9d P 64ad8799c8df4b7589869baffb93b68d [term 1 FOLLOWER]: Refusing update from remote peer a5d66df07715448fab64928f5bc8299a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:33.864558 20421 raft_consensus.cc:482] T c45ceab3a36d4225aed74654ed530c9d P 64ad8799c8df4b7589869baffb93b68d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:33.864774 20421 raft_consensus.cc:504] T c45ceab3a36d4225aed74654ed530c9d P 64ad8799c8df4b7589869baffb93b68d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.865875 20421 leader_election.cc:288] T c45ceab3a36d4225aed74654ed530c9d P 64ad8799c8df4b7589869baffb93b68d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a5d66df07715448fab64928f5bc8299a (127.16.94.130:39295), fceaf96557de46c3986c306fa4ac3b9e (127.16.94.131:39581)
I20240503 14:13:33.866977 20258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c45ceab3a36d4225aed74654ed530c9d" candidate_uuid: "64ad8799c8df4b7589869baffb93b68d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5d66df07715448fab64928f5bc8299a" is_pre_election: true
I20240503 14:13:33.869477 20177 raft_consensus.cc:1261] T 904450ab5e8b4250b7af18b98d2b0985 P 64ad8799c8df4b7589869baffb93b68d [term 1 FOLLOWER]: Refusing update from remote peer a5d66df07715448fab64928f5bc8299a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:33.869922 20482 raft_consensus.cc:482] T 904450ab5e8b4250b7af18b98d2b0985 P 64ad8799c8df4b7589869baffb93b68d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:33.870092 20482 raft_consensus.cc:504] T 904450ab5e8b4250b7af18b98d2b0985 P 64ad8799c8df4b7589869baffb93b68d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.871134 20482 leader_election.cc:288] T 904450ab5e8b4250b7af18b98d2b0985 P 64ad8799c8df4b7589869baffb93b68d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a5d66df07715448fab64928f5bc8299a (127.16.94.130:39295), fceaf96557de46c3986c306fa4ac3b9e (127.16.94.131:39581)
I20240503 14:13:33.871985 20258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904450ab5e8b4250b7af18b98d2b0985" candidate_uuid: "64ad8799c8df4b7589869baffb93b68d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5d66df07715448fab64928f5bc8299a" is_pre_election: true
I20240503 14:13:33.879271 20437 consensus_queue.cc:1035] T 44a4d2b820fc4d0da06c659fe98fe57c P a5d66df07715448fab64928f5bc8299a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20240503 14:13:33.882789 20331 raft_consensus.cc:1261] T 904450ab5e8b4250b7af18b98d2b0985 P fceaf96557de46c3986c306fa4ac3b9e [term 1 FOLLOWER]: Refusing update from remote peer a5d66df07715448fab64928f5bc8299a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:33.883299 20333 raft_consensus.cc:1261] T 5484fffa7f6b4e64ba4798e8816b6c61 P fceaf96557de46c3986c306fa4ac3b9e [term 1 FOLLOWER]: Refusing update from remote peer a5d66df07715448fab64928f5bc8299a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:33.884711 20329 raft_consensus.cc:1261] T 7a82ae1f4c66455caea352a304af2b51 P fceaf96557de46c3986c306fa4ac3b9e [term 1 FOLLOWER]: Refusing update from remote peer a5d66df07715448fab64928f5bc8299a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:33.886482 20327 raft_consensus.cc:1226] T 44a4d2b820fc4d0da06c659fe98fe57c P fceaf96557de46c3986c306fa4ac3b9e [term 2 FOLLOWER]: Rejecting Update request from peer a5d66df07715448fab64928f5bc8299a for earlier term 1. Current term is 2. Ops: [1.1-1.5]
I20240503 14:13:33.888784 20408 consensus_queue.cc:1035] T c45ceab3a36d4225aed74654ed530c9d P a5d66df07715448fab64928f5bc8299a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.034s
I20240503 14:13:33.891216 20185 raft_consensus.cc:1261] T 7a82ae1f4c66455caea352a304af2b51 P 64ad8799c8df4b7589869baffb93b68d [term 1 FOLLOWER]: Refusing update from remote peer a5d66df07715448fab64928f5bc8299a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:33.893218 20382 consensus_queue.cc:1035] T c45ceab3a36d4225aed74654ed530c9d P a5d66df07715448fab64928f5bc8299a [LEADER]: Connected to new peer: Peer: permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:13:33.893977 20382 consensus_queue.cc:1035] T 5484fffa7f6b4e64ba4798e8816b6c61 P a5d66df07715448fab64928f5bc8299a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:33.891124 20179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44a4d2b820fc4d0da06c659fe98fe57c" candidate_uuid: "fceaf96557de46c3986c306fa4ac3b9e" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64ad8799c8df4b7589869baffb93b68d"
I20240503 14:13:33.894580 20408 consensus_queue.cc:1035] T 7a82ae1f4c66455caea352a304af2b51 P a5d66df07715448fab64928f5bc8299a [LEADER]: Connected to new peer: Peer: permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20240503 14:13:33.895061 20437 consensus_queue.cc:1035] T 904450ab5e8b4250b7af18b98d2b0985 P a5d66df07715448fab64928f5bc8299a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20240503 14:13:33.911028 20479 consensus_queue.cc:1035] T 904450ab5e8b4250b7af18b98d2b0985 P a5d66df07715448fab64928f5bc8299a [LEADER]: Connected to new peer: Peer: permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.016s
W20240503 14:13:33.979601 20226 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.94.131:39581, user_credentials={real_user=slave}} blocked reactor thread for 87504.4us
I20240503 14:13:33.980520 20382 consensus_queue.cc:1035] T 7a82ae1f4c66455caea352a304af2b51 P a5d66df07715448fab64928f5bc8299a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:33.981468 20382 consensus_queue.cc:1046] T 44a4d2b820fc4d0da06c659fe98fe57c P a5d66df07715448fab64928f5bc8299a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:13:33.982208 20479 raft_consensus.cc:3032] T 44a4d2b820fc4d0da06c659fe98fe57c P a5d66df07715448fab64928f5bc8299a [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:13:33.982293 20479 raft_consensus.cc:729] T 44a4d2b820fc4d0da06c659fe98fe57c P a5d66df07715448fab64928f5bc8299a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a5d66df07715448fab64928f5bc8299a, State: Running, Role: LEADER
I20240503 14:13:33.982530 20479 consensus_queue.cc:260] T 44a4d2b820fc4d0da06c659fe98fe57c P a5d66df07715448fab64928f5bc8299a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:33.982848 20479 raft_consensus.cc:3037] T 44a4d2b820fc4d0da06c659fe98fe57c P a5d66df07715448fab64928f5bc8299a [term 1 FOLLOWER]: Advancing to term 2
W20240503 14:13:34.021725 20224 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.94.129:37429, user_credentials={real_user=slave}} blocked reactor thread for 121517us
W20240503 14:13:34.048027 20301 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.94.130:39295, user_credentials={real_user=slave}} blocked reactor thread for 199003us
I20240503 14:13:34.062157 20334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c45ceab3a36d4225aed74654ed530c9d" candidate_uuid: "64ad8799c8df4b7589869baffb93b68d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fceaf96557de46c3986c306fa4ac3b9e" is_pre_election: true
I20240503 14:13:34.064787 20455 mvcc.cc:205] Tried to move back new op lower bound from 7023598033701826560 to 7023598032869355520. Current Snapshot: MvccSnapshot[applied={T|T < 7023598033681899520}]
I20240503 14:13:34.069180 20330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904450ab5e8b4250b7af18b98d2b0985" candidate_uuid: "64ad8799c8df4b7589869baffb93b68d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fceaf96557de46c3986c306fa4ac3b9e" is_pre_election: true
I20240503 14:13:34.073982 20260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44a4d2b820fc4d0da06c659fe98fe57c" candidate_uuid: "fceaf96557de46c3986c306fa4ac3b9e" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5d66df07715448fab64928f5bc8299a"
I20240503 14:13:34.083870 20260 raft_consensus.cc:2396] T 44a4d2b820fc4d0da06c659fe98fe57c P a5d66df07715448fab64928f5bc8299a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate fceaf96557de46c3986c306fa4ac3b9e for term 2 because replica has last-logged OpId of term: 1 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20240503 14:13:34.083953 20150 leader_election.cc:302] T c45ceab3a36d4225aed74654ed530c9d P 64ad8799c8df4b7589869baffb93b68d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 64ad8799c8df4b7589869baffb93b68d; no voters: a5d66df07715448fab64928f5bc8299a, fceaf96557de46c3986c306fa4ac3b9e
I20240503 14:13:34.084496 20150 leader_election.cc:302] T 904450ab5e8b4250b7af18b98d2b0985 P 64ad8799c8df4b7589869baffb93b68d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 64ad8799c8df4b7589869baffb93b68d; no voters: a5d66df07715448fab64928f5bc8299a, fceaf96557de46c3986c306fa4ac3b9e
I20240503 14:13:34.087652 20301 leader_election.cc:302] T 44a4d2b820fc4d0da06c659fe98fe57c P fceaf96557de46c3986c306fa4ac3b9e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fceaf96557de46c3986c306fa4ac3b9e; no voters: 64ad8799c8df4b7589869baffb93b68d, a5d66df07715448fab64928f5bc8299a
I20240503 14:13:34.088291 20506 raft_consensus.cc:2726] T 44a4d2b820fc4d0da06c659fe98fe57c P fceaf96557de46c3986c306fa4ac3b9e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240503 14:13:34.139621 20260 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3054a11a8da4470ea5241b92d164d6a8"
dest_uuid: "a5d66df07715448fab64928f5bc8299a"
mode: GRACEFUL
new_leader_uuid: "fceaf96557de46c3986c306fa4ac3b9e"
from {username='slave'} at 127.0.0.1:38684
I20240503 14:13:34.166599 20482 raft_consensus.cc:2726] T 904450ab5e8b4250b7af18b98d2b0985 P 64ad8799c8df4b7589869baffb93b68d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:13:34.178547 20260 raft_consensus.cc:595] T 3054a11a8da4470ea5241b92d164d6a8 P a5d66df07715448fab64928f5bc8299a [term 1 LEADER]: Received request to transfer leadership to TS fceaf96557de46c3986c306fa4ac3b9e
I20240503 14:13:34.184242 20421 raft_consensus.cc:2726] T c45ceab3a36d4225aed74654ed530c9d P 64ad8799c8df4b7589869baffb93b68d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:13:34.187597 20260 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5484fffa7f6b4e64ba4798e8816b6c61"
dest_uuid: "a5d66df07715448fab64928f5bc8299a"
mode: GRACEFUL
new_leader_uuid: "fceaf96557de46c3986c306fa4ac3b9e"
from {username='slave'} at 127.0.0.1:38684
I20240503 14:13:34.187845 20260 raft_consensus.cc:595] T 5484fffa7f6b4e64ba4798e8816b6c61 P a5d66df07715448fab64928f5bc8299a [term 1 LEADER]: Received request to transfer leadership to TS fceaf96557de46c3986c306fa4ac3b9e
I20240503 14:13:34.189029 20260 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7a82ae1f4c66455caea352a304af2b51"
dest_uuid: "a5d66df07715448fab64928f5bc8299a"
mode: GRACEFUL
new_leader_uuid: "64ad8799c8df4b7589869baffb93b68d"
from {username='slave'} at 127.0.0.1:38684
I20240503 14:13:34.189216 20260 raft_consensus.cc:595] T 7a82ae1f4c66455caea352a304af2b51 P a5d66df07715448fab64928f5bc8299a [term 1 LEADER]: Received request to transfer leadership to TS 64ad8799c8df4b7589869baffb93b68d
I20240503 14:13:34.190479 20260 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d4743d5a73c54086b277f4c4b1a248fd"
dest_uuid: "a5d66df07715448fab64928f5bc8299a"
mode: GRACEFUL
new_leader_uuid: "64ad8799c8df4b7589869baffb93b68d"
from {username='slave'} at 127.0.0.1:38684
I20240503 14:13:34.190651 20260 raft_consensus.cc:595] T d4743d5a73c54086b277f4c4b1a248fd P a5d66df07715448fab64928f5bc8299a [term 1 LEADER]: Received request to transfer leadership to TS 64ad8799c8df4b7589869baffb93b68d
I20240503 14:13:34.192296 20136 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 4
I20240503 14:13:34.192575 20136 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:13:34.193284 20479 raft_consensus.cc:979] T 5484fffa7f6b4e64ba4798e8816b6c61 P a5d66df07715448fab64928f5bc8299a: : Instructing follower fceaf96557de46c3986c306fa4ac3b9e to start an election
I20240503 14:13:34.193693 20498 raft_consensus.cc:1067] T 5484fffa7f6b4e64ba4798e8816b6c61 P a5d66df07715448fab64928f5bc8299a [term 1 LEADER]: Signalling peer fceaf96557de46c3986c306fa4ac3b9e to start an election
I20240503 14:13:34.194053 20514 mvcc.cc:205] Tried to move back new op lower bound from 7023598033674489856 to 7023598031979618304. Current Snapshot: MvccSnapshot[applied={T|T < 7023598033674489856}]
I20240503 14:13:34.194725 20329 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5484fffa7f6b4e64ba4798e8816b6c61"
dest_uuid: "fceaf96557de46c3986c306fa4ac3b9e"
from {username='slave'} at 127.0.0.1:56216
I20240503 14:13:34.200928 20329 raft_consensus.cc:482] T 5484fffa7f6b4e64ba4798e8816b6c61 P fceaf96557de46c3986c306fa4ac3b9e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:13:34.201122 20329 raft_consensus.cc:3037] T 5484fffa7f6b4e64ba4798e8816b6c61 P fceaf96557de46c3986c306fa4ac3b9e [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:34.204350 20329 raft_consensus.cc:504] T 5484fffa7f6b4e64ba4798e8816b6c61 P fceaf96557de46c3986c306fa4ac3b9e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:34.208688 20329 leader_election.cc:288] T 5484fffa7f6b4e64ba4798e8816b6c61 P fceaf96557de46c3986c306fa4ac3b9e [CANDIDATE]: Term 2 election: Requested vote from peers 64ad8799c8df4b7589869baffb93b68d (127.16.94.129:37429), a5d66df07715448fab64928f5bc8299a (127.16.94.130:39295)
I20240503 14:13:34.209224 20260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5484fffa7f6b4e64ba4798e8816b6c61" candidate_uuid: "fceaf96557de46c3986c306fa4ac3b9e" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "a5d66df07715448fab64928f5bc8299a"
I20240503 14:13:34.209412 20260 raft_consensus.cc:3032] T 5484fffa7f6b4e64ba4798e8816b6c61 P a5d66df07715448fab64928f5bc8299a [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:13:34.209481 20260 raft_consensus.cc:729] T 5484fffa7f6b4e64ba4798e8816b6c61 P a5d66df07715448fab64928f5bc8299a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a5d66df07715448fab64928f5bc8299a, State: Running, Role: LEADER
I20240503 14:13:34.209767 20260 consensus_queue.cc:260] T 5484fffa7f6b4e64ba4798e8816b6c61 P a5d66df07715448fab64928f5bc8299a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:34.210177 20260 raft_consensus.cc:3037] T 5484fffa7f6b4e64ba4798e8816b6c61 P a5d66df07715448fab64928f5bc8299a [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:34.210981 20180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5484fffa7f6b4e64ba4798e8816b6c61" candidate_uuid: "fceaf96557de46c3986c306fa4ac3b9e" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "64ad8799c8df4b7589869baffb93b68d"
I20240503 14:13:34.211126 20180 raft_consensus.cc:3037] T 5484fffa7f6b4e64ba4798e8816b6c61 P 64ad8799c8df4b7589869baffb93b68d [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:34.213030 20260 raft_consensus.cc:2454] T 5484fffa7f6b4e64ba4798e8816b6c61 P a5d66df07715448fab64928f5bc8299a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fceaf96557de46c3986c306fa4ac3b9e in term 2.
I20240503 14:13:34.213407 20301 leader_election.cc:302] T 5484fffa7f6b4e64ba4798e8816b6c61 P fceaf96557de46c3986c306fa4ac3b9e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5d66df07715448fab64928f5bc8299a, fceaf96557de46c3986c306fa4ac3b9e; no voters:
I20240503 14:13:34.213689 20496 raft_consensus.cc:2781] T 5484fffa7f6b4e64ba4798e8816b6c61 P fceaf96557de46c3986c306fa4ac3b9e [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:13:34.213804 20496 raft_consensus.cc:686] T 5484fffa7f6b4e64ba4798e8816b6c61 P fceaf96557de46c3986c306fa4ac3b9e [term 2 LEADER]: Becoming Leader. State: Replica: fceaf96557de46c3986c306fa4ac3b9e, State: Running, Role: LEADER
I20240503 14:13:34.214059 20496 consensus_queue.cc:237] T 5484fffa7f6b4e64ba4798e8816b6c61 P fceaf96557de46c3986c306fa4ac3b9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:34.214686 20180 raft_consensus.cc:2454] T 5484fffa7f6b4e64ba4798e8816b6c61 P 64ad8799c8df4b7589869baffb93b68d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fceaf96557de46c3986c306fa4ac3b9e in term 2.
I20240503 14:13:34.216581 20078 catalog_manager.cc:5484] T 5484fffa7f6b4e64ba4798e8816b6c61 P fceaf96557de46c3986c306fa4ac3b9e reported cstate change: term changed from 1 to 2, leader changed from a5d66df07715448fab64928f5bc8299a (127.16.94.130) to fceaf96557de46c3986c306fa4ac3b9e (127.16.94.131). New cstate: current_term: 2 leader_uuid: "fceaf96557de46c3986c306fa4ac3b9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } health_report { overall_health: HEALTHY } } }
I20240503 14:13:34.443722 20437 raft_consensus.cc:979] T d4743d5a73c54086b277f4c4b1a248fd P a5d66df07715448fab64928f5bc8299a: : Instructing follower 64ad8799c8df4b7589869baffb93b68d to start an election
I20240503 14:13:34.443895 20437 raft_consensus.cc:1067] T d4743d5a73c54086b277f4c4b1a248fd P a5d66df07715448fab64928f5bc8299a [term 1 LEADER]: Signalling peer 64ad8799c8df4b7589869baffb93b68d to start an election
I20240503 14:13:34.444406 20180 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d4743d5a73c54086b277f4c4b1a248fd"
dest_uuid: "64ad8799c8df4b7589869baffb93b68d"
from {username='slave'} at 127.0.0.1:54502
I20240503 14:13:34.444559 20180 raft_consensus.cc:482] T d4743d5a73c54086b277f4c4b1a248fd P 64ad8799c8df4b7589869baffb93b68d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:13:34.444644 20180 raft_consensus.cc:3037] T d4743d5a73c54086b277f4c4b1a248fd P 64ad8799c8df4b7589869baffb93b68d [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:34.446799 20180 raft_consensus.cc:504] T d4743d5a73c54086b277f4c4b1a248fd P 64ad8799c8df4b7589869baffb93b68d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:34.447273 20180 leader_election.cc:288] T d4743d5a73c54086b277f4c4b1a248fd P 64ad8799c8df4b7589869baffb93b68d [CANDIDATE]: Term 2 election: Requested vote from peers a5d66df07715448fab64928f5bc8299a (127.16.94.130:39295), fceaf96557de46c3986c306fa4ac3b9e (127.16.94.131:39581)
I20240503 14:13:34.447638 20260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4743d5a73c54086b277f4c4b1a248fd" candidate_uuid: "64ad8799c8df4b7589869baffb93b68d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "a5d66df07715448fab64928f5bc8299a"
I20240503 14:13:34.447691 20333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4743d5a73c54086b277f4c4b1a248fd" candidate_uuid: "64ad8799c8df4b7589869baffb93b68d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "fceaf96557de46c3986c306fa4ac3b9e"
I20240503 14:13:34.447790 20260 raft_consensus.cc:3032] T d4743d5a73c54086b277f4c4b1a248fd P a5d66df07715448fab64928f5bc8299a [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:13:34.447847 20260 raft_consensus.cc:729] T d4743d5a73c54086b277f4c4b1a248fd P a5d66df07715448fab64928f5bc8299a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a5d66df07715448fab64928f5bc8299a, State: Running, Role: LEADER
I20240503 14:13:34.447854 20333 raft_consensus.cc:3037] T d4743d5a73c54086b277f4c4b1a248fd P fceaf96557de46c3986c306fa4ac3b9e [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:34.448078 20260 consensus_queue.cc:260] T d4743d5a73c54086b277f4c4b1a248fd P a5d66df07715448fab64928f5bc8299a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:34.448354 20260 raft_consensus.cc:3037] T d4743d5a73c54086b277f4c4b1a248fd P a5d66df07715448fab64928f5bc8299a [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:34.450196 20333 raft_consensus.cc:2454] T d4743d5a73c54086b277f4c4b1a248fd P fceaf96557de46c3986c306fa4ac3b9e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64ad8799c8df4b7589869baffb93b68d in term 2.
I20240503 14:13:34.450275 20260 raft_consensus.cc:2454] T d4743d5a73c54086b277f4c4b1a248fd P a5d66df07715448fab64928f5bc8299a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64ad8799c8df4b7589869baffb93b68d in term 2.
I20240503 14:13:34.450546 20150 leader_election.cc:302] T d4743d5a73c54086b277f4c4b1a248fd P 64ad8799c8df4b7589869baffb93b68d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64ad8799c8df4b7589869baffb93b68d, fceaf96557de46c3986c306fa4ac3b9e; no voters:
I20240503 14:13:34.450805 20421 raft_consensus.cc:2781] T d4743d5a73c54086b277f4c4b1a248fd P 64ad8799c8df4b7589869baffb93b68d [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:13:34.450923 20421 raft_consensus.cc:686] T d4743d5a73c54086b277f4c4b1a248fd P 64ad8799c8df4b7589869baffb93b68d [term 2 LEADER]: Becoming Leader. State: Replica: 64ad8799c8df4b7589869baffb93b68d, State: Running, Role: LEADER
I20240503 14:13:34.451140 20421 consensus_queue.cc:237] T d4743d5a73c54086b277f4c4b1a248fd P 64ad8799c8df4b7589869baffb93b68d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:34.453078 20078 catalog_manager.cc:5484] T d4743d5a73c54086b277f4c4b1a248fd P 64ad8799c8df4b7589869baffb93b68d reported cstate change: term changed from 1 to 2, leader changed from a5d66df07715448fab64928f5bc8299a (127.16.94.130) to 64ad8799c8df4b7589869baffb93b68d (127.16.94.129). New cstate: current_term: 2 leader_uuid: "64ad8799c8df4b7589869baffb93b68d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:34.576854 20497 raft_consensus.cc:979] T 3054a11a8da4470ea5241b92d164d6a8 P a5d66df07715448fab64928f5bc8299a: : Instructing follower fceaf96557de46c3986c306fa4ac3b9e to start an election
I20240503 14:13:34.577005 20497 raft_consensus.cc:1067] T 3054a11a8da4470ea5241b92d164d6a8 P a5d66df07715448fab64928f5bc8299a [term 1 LEADER]: Signalling peer fceaf96557de46c3986c306fa4ac3b9e to start an election
I20240503 14:13:34.577576 20333 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3054a11a8da4470ea5241b92d164d6a8"
dest_uuid: "fceaf96557de46c3986c306fa4ac3b9e"
from {username='slave'} at 127.0.0.1:56216
I20240503 14:13:34.577795 20333 raft_consensus.cc:482] T 3054a11a8da4470ea5241b92d164d6a8 P fceaf96557de46c3986c306fa4ac3b9e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:13:34.577883 20333 raft_consensus.cc:3037] T 3054a11a8da4470ea5241b92d164d6a8 P fceaf96557de46c3986c306fa4ac3b9e [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:34.579840 20333 raft_consensus.cc:504] T 3054a11a8da4470ea5241b92d164d6a8 P fceaf96557de46c3986c306fa4ac3b9e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:34.580397 20333 leader_election.cc:288] T 3054a11a8da4470ea5241b92d164d6a8 P fceaf96557de46c3986c306fa4ac3b9e [CANDIDATE]: Term 2 election: Requested vote from peers 64ad8799c8df4b7589869baffb93b68d (127.16.94.129:37429), a5d66df07715448fab64928f5bc8299a (127.16.94.130:39295)
I20240503 14:13:34.580772 20180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3054a11a8da4470ea5241b92d164d6a8" candidate_uuid: "fceaf96557de46c3986c306fa4ac3b9e" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "64ad8799c8df4b7589869baffb93b68d"
I20240503 14:13:34.580881 20260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3054a11a8da4470ea5241b92d164d6a8" candidate_uuid: "fceaf96557de46c3986c306fa4ac3b9e" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "a5d66df07715448fab64928f5bc8299a"
I20240503 14:13:34.580969 20180 raft_consensus.cc:3037] T 3054a11a8da4470ea5241b92d164d6a8 P 64ad8799c8df4b7589869baffb93b68d [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:34.581009 20260 raft_consensus.cc:3032] T 3054a11a8da4470ea5241b92d164d6a8 P a5d66df07715448fab64928f5bc8299a [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:13:34.581069 20260 raft_consensus.cc:729] T 3054a11a8da4470ea5241b92d164d6a8 P a5d66df07715448fab64928f5bc8299a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a5d66df07715448fab64928f5bc8299a, State: Running, Role: LEADER
I20240503 14:13:34.581337 20260 consensus_queue.cc:260] T 3054a11a8da4470ea5241b92d164d6a8 P a5d66df07715448fab64928f5bc8299a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:34.581710 20260 raft_consensus.cc:3037] T 3054a11a8da4470ea5241b92d164d6a8 P a5d66df07715448fab64928f5bc8299a [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:34.583711 20180 raft_consensus.cc:2454] T 3054a11a8da4470ea5241b92d164d6a8 P 64ad8799c8df4b7589869baffb93b68d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fceaf96557de46c3986c306fa4ac3b9e in term 2.
I20240503 14:13:34.584367 20298 leader_election.cc:302] T 3054a11a8da4470ea5241b92d164d6a8 P fceaf96557de46c3986c306fa4ac3b9e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64ad8799c8df4b7589869baffb93b68d, fceaf96557de46c3986c306fa4ac3b9e; no voters:
I20240503 14:13:34.584388 20260 raft_consensus.cc:2454] T 3054a11a8da4470ea5241b92d164d6a8 P a5d66df07715448fab64928f5bc8299a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fceaf96557de46c3986c306fa4ac3b9e in term 2.
I20240503 14:13:34.584671 20506 raft_consensus.cc:2781] T 3054a11a8da4470ea5241b92d164d6a8 P fceaf96557de46c3986c306fa4ac3b9e [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:13:34.584800 20506 raft_consensus.cc:686] T 3054a11a8da4470ea5241b92d164d6a8 P fceaf96557de46c3986c306fa4ac3b9e [term 2 LEADER]: Becoming Leader. State: Replica: fceaf96557de46c3986c306fa4ac3b9e, State: Running, Role: LEADER
I20240503 14:13:34.585041 20506 consensus_queue.cc:237] T 3054a11a8da4470ea5241b92d164d6a8 P fceaf96557de46c3986c306fa4ac3b9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:34.587000 20078 catalog_manager.cc:5484] T 3054a11a8da4470ea5241b92d164d6a8 P fceaf96557de46c3986c306fa4ac3b9e reported cstate change: term changed from 1 to 2, leader changed from a5d66df07715448fab64928f5bc8299a (127.16.94.130) to fceaf96557de46c3986c306fa4ac3b9e (127.16.94.131). New cstate: current_term: 2 leader_uuid: "fceaf96557de46c3986c306fa4ac3b9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } health_report { overall_health: HEALTHY } } }
I20240503 14:13:34.682992 20180 raft_consensus.cc:1261] T 5484fffa7f6b4e64ba4798e8816b6c61 P 64ad8799c8df4b7589869baffb93b68d [term 2 FOLLOWER]: Refusing update from remote peer fceaf96557de46c3986c306fa4ac3b9e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:13:34.683512 20506 consensus_queue.cc:1035] T 5484fffa7f6b4e64ba4798e8816b6c61 P fceaf96557de46c3986c306fa4ac3b9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:34.686962 20260 raft_consensus.cc:1261] T 5484fffa7f6b4e64ba4798e8816b6c61 P a5d66df07715448fab64928f5bc8299a [term 2 FOLLOWER]: Refusing update from remote peer fceaf96557de46c3986c306fa4ac3b9e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:13:34.687680 20506 consensus_queue.cc:1035] T 5484fffa7f6b4e64ba4798e8816b6c61 P fceaf96557de46c3986c306fa4ac3b9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:34.772495 20437 raft_consensus.cc:979] T 7a82ae1f4c66455caea352a304af2b51 P a5d66df07715448fab64928f5bc8299a: : Instructing follower 64ad8799c8df4b7589869baffb93b68d to start an election
I20240503 14:13:34.772673 20437 raft_consensus.cc:1067] T 7a82ae1f4c66455caea352a304af2b51 P a5d66df07715448fab64928f5bc8299a [term 1 LEADER]: Signalling peer 64ad8799c8df4b7589869baffb93b68d to start an election
I20240503 14:13:34.773207 20180 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7a82ae1f4c66455caea352a304af2b51"
dest_uuid: "64ad8799c8df4b7589869baffb93b68d"
from {username='slave'} at 127.0.0.1:54502
I20240503 14:13:34.773391 20180 raft_consensus.cc:482] T 7a82ae1f4c66455caea352a304af2b51 P 64ad8799c8df4b7589869baffb93b68d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:13:34.773481 20180 raft_consensus.cc:3037] T 7a82ae1f4c66455caea352a304af2b51 P 64ad8799c8df4b7589869baffb93b68d [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:34.775358 20180 raft_consensus.cc:504] T 7a82ae1f4c66455caea352a304af2b51 P 64ad8799c8df4b7589869baffb93b68d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:34.775890 20180 leader_election.cc:288] T 7a82ae1f4c66455caea352a304af2b51 P 64ad8799c8df4b7589869baffb93b68d [CANDIDATE]: Term 2 election: Requested vote from peers a5d66df07715448fab64928f5bc8299a (127.16.94.130:39295), fceaf96557de46c3986c306fa4ac3b9e (127.16.94.131:39581)
I20240503 14:13:34.776264 20333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a82ae1f4c66455caea352a304af2b51" candidate_uuid: "64ad8799c8df4b7589869baffb93b68d" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "fceaf96557de46c3986c306fa4ac3b9e"
I20240503 14:13:34.776264 20260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a82ae1f4c66455caea352a304af2b51" candidate_uuid: "64ad8799c8df4b7589869baffb93b68d" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "a5d66df07715448fab64928f5bc8299a"
I20240503 14:13:34.776479 20333 raft_consensus.cc:3037] T 7a82ae1f4c66455caea352a304af2b51 P fceaf96557de46c3986c306fa4ac3b9e [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:34.776489 20260 raft_consensus.cc:3032] T 7a82ae1f4c66455caea352a304af2b51 P a5d66df07715448fab64928f5bc8299a [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:13:34.776567 20260 raft_consensus.cc:729] T 7a82ae1f4c66455caea352a304af2b51 P a5d66df07715448fab64928f5bc8299a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a5d66df07715448fab64928f5bc8299a, State: Running, Role: LEADER
I20240503 14:13:34.776814 20260 consensus_queue.cc:260] T 7a82ae1f4c66455caea352a304af2b51 P a5d66df07715448fab64928f5bc8299a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:34.777160 20260 raft_consensus.cc:3037] T 7a82ae1f4c66455caea352a304af2b51 P a5d66df07715448fab64928f5bc8299a [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:34.779294 20333 raft_consensus.cc:2454] T 7a82ae1f4c66455caea352a304af2b51 P fceaf96557de46c3986c306fa4ac3b9e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64ad8799c8df4b7589869baffb93b68d in term 2.
I20240503 14:13:34.779702 20260 raft_consensus.cc:2454] T 7a82ae1f4c66455caea352a304af2b51 P a5d66df07715448fab64928f5bc8299a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64ad8799c8df4b7589869baffb93b68d in term 2.
I20240503 14:13:34.779700 20150 leader_election.cc:302] T 7a82ae1f4c66455caea352a304af2b51 P 64ad8799c8df4b7589869baffb93b68d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64ad8799c8df4b7589869baffb93b68d, fceaf96557de46c3986c306fa4ac3b9e; no voters:
I20240503 14:13:34.779999 20421 raft_consensus.cc:2781] T 7a82ae1f4c66455caea352a304af2b51 P 64ad8799c8df4b7589869baffb93b68d [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:13:34.780112 20421 raft_consensus.cc:686] T 7a82ae1f4c66455caea352a304af2b51 P 64ad8799c8df4b7589869baffb93b68d [term 2 LEADER]: Becoming Leader. State: Replica: 64ad8799c8df4b7589869baffb93b68d, State: Running, Role: LEADER
I20240503 14:13:34.780336 20421 consensus_queue.cc:237] T 7a82ae1f4c66455caea352a304af2b51 P 64ad8799c8df4b7589869baffb93b68d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:34.782649 20078 catalog_manager.cc:5484] T 7a82ae1f4c66455caea352a304af2b51 P 64ad8799c8df4b7589869baffb93b68d reported cstate change: term changed from 1 to 2, leader changed from a5d66df07715448fab64928f5bc8299a (127.16.94.130) to 64ad8799c8df4b7589869baffb93b68d (127.16.94.129). New cstate: current_term: 2 leader_uuid: "64ad8799c8df4b7589869baffb93b68d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:34.929114 20333 raft_consensus.cc:1261] T d4743d5a73c54086b277f4c4b1a248fd P fceaf96557de46c3986c306fa4ac3b9e [term 2 FOLLOWER]: Refusing update from remote peer 64ad8799c8df4b7589869baffb93b68d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 14:13:34.929621 20482 consensus_queue.cc:1035] T d4743d5a73c54086b277f4c4b1a248fd P 64ad8799c8df4b7589869baffb93b68d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:34.932927 20260 raft_consensus.cc:1261] T d4743d5a73c54086b277f4c4b1a248fd P a5d66df07715448fab64928f5bc8299a [term 2 FOLLOWER]: Refusing update from remote peer 64ad8799c8df4b7589869baffb93b68d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 14:13:34.933588 20421 consensus_queue.cc:1035] T d4743d5a73c54086b277f4c4b1a248fd P 64ad8799c8df4b7589869baffb93b68d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:35.060178 20260 raft_consensus.cc:1261] T 3054a11a8da4470ea5241b92d164d6a8 P a5d66df07715448fab64928f5bc8299a [term 2 FOLLOWER]: Refusing update from remote peer fceaf96557de46c3986c306fa4ac3b9e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:13:35.060784 20506 consensus_queue.cc:1035] T 3054a11a8da4470ea5241b92d164d6a8 P fceaf96557de46c3986c306fa4ac3b9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:35.063505 20180 raft_consensus.cc:1261] T 3054a11a8da4470ea5241b92d164d6a8 P 64ad8799c8df4b7589869baffb93b68d [term 2 FOLLOWER]: Refusing update from remote peer fceaf96557de46c3986c306fa4ac3b9e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:13:35.064278 20496 consensus_queue.cc:1035] T 3054a11a8da4470ea5241b92d164d6a8 P fceaf96557de46c3986c306fa4ac3b9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:35.176883 20333 raft_consensus.cc:1261] T 7a82ae1f4c66455caea352a304af2b51 P fceaf96557de46c3986c306fa4ac3b9e [term 2 FOLLOWER]: Refusing update from remote peer 64ad8799c8df4b7589869baffb93b68d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:13:35.177402 20421 consensus_queue.cc:1035] T 7a82ae1f4c66455caea352a304af2b51 P 64ad8799c8df4b7589869baffb93b68d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:35.180896 20260 raft_consensus.cc:1261] T 7a82ae1f4c66455caea352a304af2b51 P a5d66df07715448fab64928f5bc8299a [term 2 FOLLOWER]: Refusing update from remote peer 64ad8799c8df4b7589869baffb93b68d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:13:35.181414 20523 consensus_queue.cc:1035] T 7a82ae1f4c66455caea352a304af2b51 P 64ad8799c8df4b7589869baffb93b68d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:35.284255 20437 raft_consensus.cc:482] T 44a4d2b820fc4d0da06c659fe98fe57c P a5d66df07715448fab64928f5bc8299a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:35.284477 20437 raft_consensus.cc:504] T 44a4d2b820fc4d0da06c659fe98fe57c P a5d66df07715448fab64928f5bc8299a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:35.285215 20437 leader_election.cc:288] T 44a4d2b820fc4d0da06c659fe98fe57c P a5d66df07715448fab64928f5bc8299a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 64ad8799c8df4b7589869baffb93b68d (127.16.94.129:37429), fceaf96557de46c3986c306fa4ac3b9e (127.16.94.131:39581)
I20240503 14:13:35.285696 20333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44a4d2b820fc4d0da06c659fe98fe57c" candidate_uuid: "a5d66df07715448fab64928f5bc8299a" candidate_term: 3 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "fceaf96557de46c3986c306fa4ac3b9e" is_pre_election: true
I20240503 14:13:35.285703 20180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44a4d2b820fc4d0da06c659fe98fe57c" candidate_uuid: "a5d66df07715448fab64928f5bc8299a" candidate_term: 3 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "64ad8799c8df4b7589869baffb93b68d" is_pre_election: true
I20240503 14:13:35.285887 20333 raft_consensus.cc:2454] T 44a4d2b820fc4d0da06c659fe98fe57c P fceaf96557de46c3986c306fa4ac3b9e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5d66df07715448fab64928f5bc8299a in term 2.
I20240503 14:13:35.286237 20226 leader_election.cc:302] T 44a4d2b820fc4d0da06c659fe98fe57c P a5d66df07715448fab64928f5bc8299a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5d66df07715448fab64928f5bc8299a, fceaf96557de46c3986c306fa4ac3b9e; no voters:
I20240503 14:13:35.286468 20437 raft_consensus.cc:2781] T 44a4d2b820fc4d0da06c659fe98fe57c P a5d66df07715448fab64928f5bc8299a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240503 14:13:35.286540 20437 raft_consensus.cc:482] T 44a4d2b820fc4d0da06c659fe98fe57c P a5d66df07715448fab64928f5bc8299a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:35.286586 20437 raft_consensus.cc:3037] T 44a4d2b820fc4d0da06c659fe98fe57c P a5d66df07715448fab64928f5bc8299a [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:13:35.288398 20437 raft_consensus.cc:504] T 44a4d2b820fc4d0da06c659fe98fe57c P a5d66df07715448fab64928f5bc8299a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:35.288972 20437 leader_election.cc:288] T 44a4d2b820fc4d0da06c659fe98fe57c P a5d66df07715448fab64928f5bc8299a [CANDIDATE]: Term 3 election: Requested vote from peers 64ad8799c8df4b7589869baffb93b68d (127.16.94.129:37429), fceaf96557de46c3986c306fa4ac3b9e (127.16.94.131:39581)
I20240503 14:13:35.289284 20180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44a4d2b820fc4d0da06c659fe98fe57c" candidate_uuid: "a5d66df07715448fab64928f5bc8299a" candidate_term: 3 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "64ad8799c8df4b7589869baffb93b68d"
I20240503 14:13:35.289433 20333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44a4d2b820fc4d0da06c659fe98fe57c" candidate_uuid: "a5d66df07715448fab64928f5bc8299a" candidate_term: 3 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "fceaf96557de46c3986c306fa4ac3b9e"
I20240503 14:13:35.289568 20333 raft_consensus.cc:3037] T 44a4d2b820fc4d0da06c659fe98fe57c P fceaf96557de46c3986c306fa4ac3b9e [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:13:35.291373 20333 raft_consensus.cc:2454] T 44a4d2b820fc4d0da06c659fe98fe57c P fceaf96557de46c3986c306fa4ac3b9e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5d66df07715448fab64928f5bc8299a in term 3.
I20240503 14:13:35.291726 20226 leader_election.cc:302] T 44a4d2b820fc4d0da06c659fe98fe57c P a5d66df07715448fab64928f5bc8299a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a5d66df07715448fab64928f5bc8299a, fceaf96557de46c3986c306fa4ac3b9e; no voters: 64ad8799c8df4b7589869baffb93b68d
I20240503 14:13:35.292007 20437 raft_consensus.cc:2781] T 44a4d2b820fc4d0da06c659fe98fe57c P a5d66df07715448fab64928f5bc8299a [term 3 FOLLOWER]: Leader election won for term 3
I20240503 14:13:35.292315 20437 raft_consensus.cc:686] T 44a4d2b820fc4d0da06c659fe98fe57c P a5d66df07715448fab64928f5bc8299a [term 3 LEADER]: Becoming Leader. State: Replica: a5d66df07715448fab64928f5bc8299a, State: Running, Role: LEADER
I20240503 14:13:35.292572 20437 consensus_queue.cc:237] T 44a4d2b820fc4d0da06c659fe98fe57c P a5d66df07715448fab64928f5bc8299a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:35.295208 20078 catalog_manager.cc:5484] T 44a4d2b820fc4d0da06c659fe98fe57c P a5d66df07715448fab64928f5bc8299a reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "a5d66df07715448fab64928f5bc8299a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:35.342454 20482 raft_consensus.cc:482] T 44a4d2b820fc4d0da06c659fe98fe57c P 64ad8799c8df4b7589869baffb93b68d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a5d66df07715448fab64928f5bc8299a)
I20240503 14:13:35.342646 20482 raft_consensus.cc:504] T 44a4d2b820fc4d0da06c659fe98fe57c P 64ad8799c8df4b7589869baffb93b68d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:35.343215 20482 leader_election.cc:288] T 44a4d2b820fc4d0da06c659fe98fe57c P 64ad8799c8df4b7589869baffb93b68d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a5d66df07715448fab64928f5bc8299a (127.16.94.130:39295), fceaf96557de46c3986c306fa4ac3b9e (127.16.94.131:39581)
I20240503 14:13:35.343664 20260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44a4d2b820fc4d0da06c659fe98fe57c" candidate_uuid: "64ad8799c8df4b7589869baffb93b68d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a5d66df07715448fab64928f5bc8299a" is_pre_election: true
I20240503 14:13:35.343648 20333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44a4d2b820fc4d0da06c659fe98fe57c" candidate_uuid: "64ad8799c8df4b7589869baffb93b68d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fceaf96557de46c3986c306fa4ac3b9e" is_pre_election: true
I20240503 14:13:35.343925 20333 raft_consensus.cc:2354] T 44a4d2b820fc4d0da06c659fe98fe57c P fceaf96557de46c3986c306fa4ac3b9e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 64ad8799c8df4b7589869baffb93b68d for earlier term 2. Current term is 3.
I20240503 14:13:35.344296 20150 leader_election.cc:398] T 44a4d2b820fc4d0da06c659fe98fe57c P 64ad8799c8df4b7589869baffb93b68d [CANDIDATE]: Term 2 pre-election: Vote denied by peer fceaf96557de46c3986c306fa4ac3b9e (127.16.94.131:39581) with higher term. Message: Invalid argument: T 44a4d2b820fc4d0da06c659fe98fe57c P fceaf96557de46c3986c306fa4ac3b9e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 64ad8799c8df4b7589869baffb93b68d for earlier term 2. Current term is 3.
I20240503 14:13:35.344362 20150 leader_election.cc:401] T 44a4d2b820fc4d0da06c659fe98fe57c P 64ad8799c8df4b7589869baffb93b68d [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20240503 14:13:35.344549 20482 raft_consensus.cc:3037] T 44a4d2b820fc4d0da06c659fe98fe57c P 64ad8799c8df4b7589869baffb93b68d [term 1 FOLLOWER]: Advancing to term 3
I20240503 14:13:35.346626 20482 raft_consensus.cc:2726] T 44a4d2b820fc4d0da06c659fe98fe57c P 64ad8799c8df4b7589869baffb93b68d [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer fceaf96557de46c3986c306fa4ac3b9e (127.16.94.131:39581) with higher term. Message: Invalid argument: T 44a4d2b820fc4d0da06c659fe98fe57c P fceaf96557de46c3986c306fa4ac3b9e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 64ad8799c8df4b7589869baffb93b68d for earlier term 2. Current term is 3.
I20240503 14:13:35.882203 20180 raft_consensus.cc:1261] T 44a4d2b820fc4d0da06c659fe98fe57c P 64ad8799c8df4b7589869baffb93b68d [term 3 FOLLOWER]: Refusing update from remote peer a5d66df07715448fab64928f5bc8299a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20240503 14:13:35.882699 20437 consensus_queue.cc:1035] T 44a4d2b820fc4d0da06c659fe98fe57c P a5d66df07715448fab64928f5bc8299a [LEADER]: Connected to new peer: Peer: permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:35.888487 20534 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:38674 (ReqId={client: 81059defe2bb4752bf7b85df82320ba1, seq_no=15, attempt_no=0}) took 2241 ms. Trace:
I20240503 14:13:35.889669 20537 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:38674 (ReqId={client: 81059defe2bb4752bf7b85df82320ba1, seq_no=8, attempt_no=0}) took 2252 ms. Trace:
I20240503 14:13:35.888756 20534 rpcz_store.cc:276] 0503 14:13:33.646741 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:13:33.650520 (+ 3779us) service_pool.cc:225] Handling call
0503 14:13:35.888439 (+2237919us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:13:33.679318 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 44a4d2b820fc4d0da06c659fe98fe57c
0503 14:13:33.679415 (+ 97us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:13:33.679420 (+ 5us) write_op.cc:435] Acquired schema lock
0503 14:13:33.679422 (+ 2us) tablet.cc:660] Decoding operations
0503 14:13:33.679542 (+ 120us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:13:33.679556 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0503 14:13:33.679563 (+ 7us) tablet.cc:683] Acquiring locks for 5 operations
0503 14:13:33.687233 (+ 7670us) tablet.cc:699] Row locks acquired
0503 14:13:33.687237 (+ 4us) write_op.cc:260] PREPARE: finished
0503 14:13:33.687285 (+ 48us) write_op.cc:270] Start()
0503 14:13:33.687320 (+ 35us) write_op.cc:276] Timestamp: P: 1714745613687274 usec, L: 0
0503 14:13:33.687326 (+ 6us) op_driver.cc:331] REPLICATION: starting
0503 14:13:33.687559 (+ 233us) log.cc:842] Serialized 519 byte log entry
0503 14:13:35.886911 (+2199352us) op_driver.cc:444] REPLICATION: finished
0503 14:13:35.887696 (+ 785us) write_op.cc:301] APPLY: starting
0503 14:13:35.887708 (+ 12us) tablet.cc:1365] starting BulkCheckPresence
0503 14:13:35.887785 (+ 77us) tablet.cc:1368] finished BulkCheckPresence
0503 14:13:35.887829 (+ 44us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:13:35.888019 (+ 190us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:13:35.888047 (+ 28us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0503 14:13:35.888053 (+ 6us) write_op.cc:312] APPLY: finished
0503 14:13:35.888110 (+ 57us) log.cc:842] Serialized 46 byte log entry
0503 14:13:35.888332 (+ 222us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:13:35.888348 (+ 16us) write_op.cc:454] Released schema lock
0503 14:13:35.888353 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":738,"mutex_wait_us":37567,"num_ops":5,"prepare.queue_time_us":28319,"prepare.run_cpu_time_us":915,"prepare.run_wall_time_us":8403,"raft.queue_time_us":297,"raft.run_cpu_time_us":426,"raft.run_wall_time_us":8628,"replication_time_us":2199561,"thread_start_us":96814,"threads_started":2}]]}
I20240503 14:13:35.889827 20537 rpcz_store.cc:276] 0503 14:13:33.636832 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:13:33.636876 (+ 44us) service_pool.cc:225] Handling call
0503 14:13:35.889636 (+2252760us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:13:33.769632 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 44a4d2b820fc4d0da06c659fe98fe57c
0503 14:13:33.770889 (+ 1257us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:13:33.770898 (+ 9us) write_op.cc:435] Acquired schema lock
0503 14:13:33.770901 (+ 3us) tablet.cc:660] Decoding operations
0503 14:13:33.771007 (+ 106us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:13:33.771021 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0503 14:13:33.771029 (+ 8us) tablet.cc:683] Acquiring locks for 1 operations
0503 14:13:33.771090 (+ 61us) tablet.cc:699] Row locks acquired
0503 14:13:33.771095 (+ 5us) write_op.cc:260] PREPARE: finished
0503 14:13:33.771141 (+ 46us) write_op.cc:270] Start()
0503 14:13:33.771181 (+ 40us) write_op.cc:276] Timestamp: P: 1714745613771129 usec, L: 0
0503 14:13:33.771187 (+ 6us) op_driver.cc:331] REPLICATION: starting
0503 14:13:33.771430 (+ 243us) log.cc:842] Serialized 345 byte log entry
0503 14:13:35.887654 (+2116224us) op_driver.cc:444] REPLICATION: finished
0503 14:13:35.889215 (+ 1561us) write_op.cc:301] APPLY: starting
0503 14:13:35.889227 (+ 12us) tablet.cc:1365] starting BulkCheckPresence
0503 14:13:35.889262 (+ 35us) tablet.cc:1368] finished BulkCheckPresence
0503 14:13:35.889285 (+ 23us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:13:35.889356 (+ 71us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:13:35.889381 (+ 25us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0503 14:13:35.889388 (+ 7us) write_op.cc:312] APPLY: finished
0503 14:13:35.889465 (+ 77us) log.cc:842] Serialized 22 byte log entry
0503 14:13:35.889527 (+ 62us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:13:35.889541 (+ 14us) write_op.cc:454] Released schema lock
0503 14:13:35.889546 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1498,"num_ops":1,"prepare.queue_time_us":40911,"prepare.run_cpu_time_us":821,"prepare.run_wall_time_us":2311,"replication_time_us":2116443,"thread_start_us":121126,"threads_started":2}]]}
I20240503 14:13:35.892760 20537 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:38674 (ReqId={client: 81059defe2bb4752bf7b85df82320ba1, seq_no=43, attempt_no=0}) took 2148 ms. Trace:
I20240503 14:13:35.892844 20537 rpcz_store.cc:276] 0503 14:13:33.744374 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:13:33.746566 (+ 2192us) service_pool.cc:225] Handling call
0503 14:13:35.892744 (+2146178us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:13:33.807920 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 44a4d2b820fc4d0da06c659fe98fe57c
0503 14:13:33.808036 (+ 116us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:13:33.808043 (+ 7us) write_op.cc:435] Acquired schema lock
0503 14:13:33.808045 (+ 2us) tablet.cc:660] Decoding operations
0503 14:13:33.813938 (+ 5893us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:13:33.813949 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0503 14:13:33.813957 (+ 8us) tablet.cc:683] Acquiring locks for 8 operations
0503 14:13:33.825540 (+ 11583us) tablet.cc:699] Row locks acquired
0503 14:13:33.825543 (+ 3us) write_op.cc:260] PREPARE: finished
0503 14:13:33.825584 (+ 41us) write_op.cc:270] Start()
0503 14:13:33.825615 (+ 31us) write_op.cc:276] Timestamp: P: 1714745613825574 usec, L: 0
0503 14:13:33.825618 (+ 3us) op_driver.cc:331] REPLICATION: starting
0503 14:13:33.825989 (+ 371us) log.cc:842] Serialized 649 byte log entry
0503 14:13:35.891216 (+2065227us) op_driver.cc:444] REPLICATION: finished
0503 14:13:35.892301 (+ 1085us) write_op.cc:301] APPLY: starting
0503 14:13:35.892311 (+ 10us) tablet.cc:1365] starting BulkCheckPresence
0503 14:13:35.892349 (+ 38us) tablet.cc:1368] finished BulkCheckPresence
0503 14:13:35.892370 (+ 21us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:13:35.892525 (+ 155us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:13:35.892546 (+ 21us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0503 14:13:35.892550 (+ 4us) write_op.cc:312] APPLY: finished
0503 14:13:35.892597 (+ 47us) log.cc:842] Serialized 64 byte log entry
0503 14:13:35.892656 (+ 59us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:13:35.892675 (+ 19us) write_op.cc:454] Released schema lock
0503 14:13:35.892680 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1054,"num_ops":8,"prepare.queue_time_us":41,"prepare.run_cpu_time_us":892,"prepare.run_wall_time_us":18144,"replication_time_us":2065582,"threads_started":1}]]}
I20240503 14:13:35.891889 20538 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:38674 (ReqId={client: 81059defe2bb4752bf7b85df82320ba1, seq_no=24, attempt_no=0}) took 2236 ms. Trace:
I20240503 14:13:35.893687 20538 rpcz_store.cc:276] 0503 14:13:33.654989 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:13:33.698035 (+ 43046us) service_pool.cc:225] Handling call
0503 14:13:35.891874 (+2193839us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 14:13:33.783511 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 44a4d2b820fc4d0da06c659fe98fe57c
0503 14:13:33.783641 (+ 130us) write_op.cc:432] Acquiring schema lock in shared mode
0503 14:13:33.783648 (+ 7us) write_op.cc:435] Acquired schema lock
0503 14:13:33.783652 (+ 4us) tablet.cc:660] Decoding operations
0503 14:13:33.783780 (+ 128us) write_op.cc:620] Acquiring the partition lock for write op
0503 14:13:33.783791 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0503 14:13:33.783798 (+ 7us) tablet.cc:683] Acquiring locks for 4 operations
0503 14:13:33.783917 (+ 119us) tablet.cc:699] Row locks acquired
0503 14:13:33.783921 (+ 4us) write_op.cc:260] PREPARE: finished
0503 14:13:33.783963 (+ 42us) write_op.cc:270] Start()
0503 14:13:33.784001 (+ 38us) write_op.cc:276] Timestamp: P: 1714745613783952 usec, L: 0
0503 14:13:33.784007 (+ 6us) op_driver.cc:331] REPLICATION: starting
0503 14:13:33.784182 (+ 175us) log.cc:842] Serialized 475 byte log entry
0503 14:13:35.889135 (+2104953us) op_driver.cc:444] REPLICATION: finished
0503 14:13:35.891271 (+ 2136us) write_op.cc:301] APPLY: starting
0503 14:13:35.891283 (+ 12us) tablet.cc:1365] starting BulkCheckPresence
0503 14:13:35.891327 (+ 44us) tablet.cc:1368] finished BulkCheckPresence
0503 14:13:35.891382 (+ 55us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 14:13:35.891565 (+ 183us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 14:13:35.891595 (+ 30us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0503 14:13:35.891604 (+ 9us) write_op.cc:312] APPLY: finished
0503 14:13:35.891677 (+ 73us) log.cc:842] Serialized 40 byte log entry
0503 14:13:35.891733 (+ 56us) write_op.cc:489] Releasing partition, row and schema locks
0503 14:13:35.891750 (+ 17us) write_op.cc:454] Released schema lock
0503 14:13:35.891755 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":2077,"num_ops":4,"prepare.queue_time_us":36605,"prepare.run_cpu_time_us":746,"prepare.run_wall_time_us":746,"replication_time_us":2105110,"thread_start_us":2007,"threads_started":1}]]}
I20240503 14:13:35.898681 20333 raft_consensus.cc:1261] T 44a4d2b820fc4d0da06c659fe98fe57c P fceaf96557de46c3986c306fa4ac3b9e [term 3 FOLLOWER]: Refusing update from remote peer a5d66df07715448fab64928f5bc8299a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20240503 14:13:35.899174 20437 consensus_queue.cc:1035] T 44a4d2b820fc4d0da06c659fe98fe57c P a5d66df07715448fab64928f5bc8299a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:13:35.910305 20240 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38674: Illegal state: replica a5d66df07715448fab64928f5bc8299a is not leader of this config: current role FOLLOWER
W20240503 14:13:35.915361 20234 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38674: Illegal state: replica a5d66df07715448fab64928f5bc8299a is not leader of this config: current role FOLLOWER
W20240503 14:13:35.917599 20161 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54526: Illegal state: replica 64ad8799c8df4b7589869baffb93b68d is not leader of this config: current role FOLLOWER
W20240503 14:13:35.925781 20234 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38674: Illegal state: replica a5d66df07715448fab64928f5bc8299a is not leader of this config: current role FOLLOWER
W20240503 14:13:35.929121 20162 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54526: Illegal state: replica 64ad8799c8df4b7589869baffb93b68d is not leader of this config: current role FOLLOWER
W20240503 14:13:35.930326 20160 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54526: Illegal state: replica 64ad8799c8df4b7589869baffb93b68d is not leader of this config: current role FOLLOWER
W20240503 14:13:35.931208 20163 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54526: Illegal state: replica 64ad8799c8df4b7589869baffb93b68d is not leader of this config: current role FOLLOWER
W20240503 14:13:35.934651 20161 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54526: Illegal state: replica 64ad8799c8df4b7589869baffb93b68d is not leader of this config: current role FOLLOWER
W20240503 14:13:35.938019 20239 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38674: Illegal state: replica a5d66df07715448fab64928f5bc8299a is not leader of this config: current role FOLLOWER
W20240503 14:13:35.947654 20234 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38674: Illegal state: replica a5d66df07715448fab64928f5bc8299a is not leader of this config: current role FOLLOWER
W20240503 14:13:35.956104 20240 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38674: Illegal state: replica a5d66df07715448fab64928f5bc8299a is not leader of this config: current role FOLLOWER
W20240503 14:13:35.973023 20234 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38674: Illegal state: replica a5d66df07715448fab64928f5bc8299a is not leader of this config: current role FOLLOWER
W20240503 14:13:35.974069 20240 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38674: Illegal state: replica a5d66df07715448fab64928f5bc8299a is not leader of this config: current role FOLLOWER
I20240503 14:13:36.194288 20136 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
W20240503 14:13:36.285573 20587 log.cc:924] Time spent T d4743d5a73c54086b277f4c4b1a248fd P fceaf96557de46c3986c306fa4ac3b9e: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
I20240503 14:13:36.286368 20136 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:13:36.286633 20136 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
W20240503 14:13:36.520973 20150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.94.131:39581, user_credentials={real_user=slave}} blocked reactor thread for 99658.4us
I20240503 14:13:37.109617 16762 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:13:37.118676 20612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:37.119112 20613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:37.119551 20615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:13:37.120491 16762 server_base.cc:1034] running on GCE node
I20240503 14:13:37.120738 16762 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:13:37.120787 16762 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:13:37.120842 16762 hybrid_clock.cc:648] HybridClock initialized: now 1714745617120839 us; error 0 us; skew 500 ppm
I20240503 14:13:37.120963 16762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:13:37.127663 16762 webserver.cc:460] Webserver started at http://127.16.94.132:42785/ using document root <none> and password file <none>
I20240503 14:13:37.127876 16762 fs_manager.cc:359] Metadata directory not provided
I20240503 14:13:37.127947 16762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:13:37.128053 16762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:13:37.128631 16762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714745515241994-16762-0/minicluster-data/ts-3-root/instance:
uuid: "c4d0f91a34804de99e3a2a255cd06aed"
format_stamp: "Formatted at 2024-05-03 14:13:37 on dist-test-slave-v27g"
I20240503 14:13:37.131175 16762 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 14:13:37.132712 20620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:37.133059 16762 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240503 14:13:37.133147 16762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714745515241994-16762-0/minicluster-data/ts-3-root
uuid: "c4d0f91a34804de99e3a2a255cd06aed"
format_stamp: "Formatted at 2024-05-03 14:13:37 on dist-test-slave-v27g"
I20240503 14:13:37.133255 16762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714745515241994-16762-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714745515241994-16762-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714745515241994-16762-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:13:37.142999 16762 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:13:37.143455 16762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:13:37.143998 16762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:13:37.144773 16762 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:13:37.144829 16762 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:37.144909 16762 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:13:37.144953 16762 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:37.159912 16762 rpc_server.cc:307] RPC server started. Bound to: 127.16.94.132:41945
I20240503 14:13:37.159945 20682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.94.132:41945 every 8 connection(s)
I20240503 14:13:37.161630 16762 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:13:37.164917 20687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:37.165841 20688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:13:37.167860 20683 heartbeater.cc:347] Connected to a master server at 127.16.94.190:39321
I20240503 14:13:37.167989 20683 heartbeater.cc:464] Registering TS with master...
I20240503 14:13:37.168334 20683 heartbeater.cc:511] Master 127.16.94.190:39321 requested a full tablet report, sending...
I20240503 14:13:37.168665 16762 server_base.cc:1034] running on GCE node
I20240503 14:13:37.169160 20078 ts_manager.cc:194] Registered new tserver with Master: c4d0f91a34804de99e3a2a255cd06aed (127.16.94.132:41945)
W20240503 14:13:37.169169 20690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:13:37.169534 16762 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:13:37.169591 16762 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:13:37.169615 16762 hybrid_clock.cc:648] HybridClock initialized: now 1714745617169612 us; error 0 us; skew 500 ppm
I20240503 14:13:37.169750 16762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:13:37.170188 20078 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58848
I20240503 14:13:37.171180 16762 webserver.cc:460] Webserver started at http://127.16.94.133:38315/ using document root <none> and password file <none>
I20240503 14:13:37.171365 16762 fs_manager.cc:359] Metadata directory not provided
I20240503 14:13:37.171422 16762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:13:37.171571 16762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:13:37.172153 16762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714745515241994-16762-0/minicluster-data/ts-4-root/instance:
uuid: "f8c99fddf5974401b64aa8601605b2b2"
format_stamp: "Formatted at 2024-05-03 14:13:37 on dist-test-slave-v27g"
I20240503 14:13:37.174396 16762 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 14:13:37.175808 20695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:37.176122 16762 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 14:13:37.176214 16762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714745515241994-16762-0/minicluster-data/ts-4-root
uuid: "f8c99fddf5974401b64aa8601605b2b2"
format_stamp: "Formatted at 2024-05-03 14:13:37 on dist-test-slave-v27g"
I20240503 14:13:37.176306 16762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714745515241994-16762-0/minicluster-data/ts-4-root
metadata directory: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714745515241994-16762-0/minicluster-data/ts-4-root
1 data directories: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714745515241994-16762-0/minicluster-data/ts-4-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:13:37.189736 16762 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:13:37.190222 16762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:13:37.190680 16762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:13:37.191416 16762 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:13:37.191469 16762 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:37.191532 16762 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:13:37.191567 16762 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:37.207448 16762 rpc_server.cc:307] RPC server started. Bound to: 127.16.94.133:42559
I20240503 14:13:37.207477 20758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.94.133:42559 every 8 connection(s)
I20240503 14:13:37.209172 16762 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:13:37.212736 20762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:37.213232 20763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:37.213721 20765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:13:37.214299 16762 server_base.cc:1034] running on GCE node
I20240503 14:13:37.214578 16762 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:13:37.214697 16762 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:13:37.214769 16762 hybrid_clock.cc:648] HybridClock initialized: now 1714745617214739 us; error 0 us; skew 500 ppm
I20240503 14:13:37.214926 16762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:13:37.215332 20759 heartbeater.cc:347] Connected to a master server at 127.16.94.190:39321
I20240503 14:13:37.215476 20759 heartbeater.cc:464] Registering TS with master...
I20240503 14:13:37.215905 20759 heartbeater.cc:511] Master 127.16.94.190:39321 requested a full tablet report, sending...
I20240503 14:13:37.216300 16762 webserver.cc:460] Webserver started at http://127.16.94.134:32921/ using document root <none> and password file <none>
I20240503 14:13:37.216573 16762 fs_manager.cc:359] Metadata directory not provided
I20240503 14:13:37.216682 16762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:13:37.216773 20078 ts_manager.cc:194] Registered new tserver with Master: f8c99fddf5974401b64aa8601605b2b2 (127.16.94.133:42559)
I20240503 14:13:37.216851 16762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:13:37.217470 16762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714745515241994-16762-0/minicluster-data/ts-5-root/instance:
uuid: "d6bcbc9fecfb44fa99a81917ea2d559a"
format_stamp: "Formatted at 2024-05-03 14:13:37 on dist-test-slave-v27g"
I20240503 14:13:37.217520 20078 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58858
I20240503 14:13:37.220005 16762 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20240503 14:13:37.221460 20770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:37.221856 16762 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240503 14:13:37.221964 16762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714745515241994-16762-0/minicluster-data/ts-5-root
uuid: "d6bcbc9fecfb44fa99a81917ea2d559a"
format_stamp: "Formatted at 2024-05-03 14:13:37 on dist-test-slave-v27g"
I20240503 14:13:37.222092 16762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714745515241994-16762-0/minicluster-data/ts-5-root
metadata directory: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714745515241994-16762-0/minicluster-data/ts-5-root
1 data directories: /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714745515241994-16762-0/minicluster-data/ts-5-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:13:37.233398 16762 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:13:37.233975 16762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:13:37.234458 16762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:13:37.235236 16762 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:13:37.235301 16762 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:37.235366 16762 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:13:37.235409 16762 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:37.249331 16762 rpc_server.cc:307] RPC server started. Bound to: 127.16.94.134:34983
I20240503 14:13:37.249403 20832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.94.134:34983 every 8 connection(s)
I20240503 14:13:37.256516 20833 heartbeater.cc:347] Connected to a master server at 127.16.94.190:39321
I20240503 14:13:37.256656 20833 heartbeater.cc:464] Registering TS with master...
I20240503 14:13:37.257056 20833 heartbeater.cc:511] Master 127.16.94.190:39321 requested a full tablet report, sending...
I20240503 14:13:37.257967 20078 ts_manager.cc:194] Registered new tserver with Master: d6bcbc9fecfb44fa99a81917ea2d559a (127.16.94.134:34983)
I20240503 14:13:37.258724 20078 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58874
I20240503 14:13:37.858475 20179 consensus_queue.cc:237] T 220ef8bab4694ace9bdf0fce06226efa P 64ad8799c8df4b7589869baffb93b68d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } attrs { replace: true } } peers { permanent_uuid: "f8c99fddf5974401b64aa8601605b2b2" member_type: NON_VOTER last_known_addr { host: "127.16.94.133" port: 42559 } attrs { promote: true } }
I20240503 14:13:37.860415 20261 raft_consensus.cc:1261] T 220ef8bab4694ace9bdf0fce06226efa P a5d66df07715448fab64928f5bc8299a [term 1 FOLLOWER]: Refusing update from remote peer 64ad8799c8df4b7589869baffb93b68d: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 14:13:37.861129 20330 raft_consensus.cc:1261] T 220ef8bab4694ace9bdf0fce06226efa P fceaf96557de46c3986c306fa4ac3b9e [term 1 FOLLOWER]: Refusing update from remote peer 64ad8799c8df4b7589869baffb93b68d: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 14:13:37.861600 20606 consensus_queue.cc:1035] T 220ef8bab4694ace9bdf0fce06226efa P 64ad8799c8df4b7589869baffb93b68d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 14:13:37.862257 20838 consensus_queue.cc:1035] T 220ef8bab4694ace9bdf0fce06226efa P 64ad8799c8df4b7589869baffb93b68d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 14:13:37.864370 20838 raft_consensus.cc:2932] T 220ef8bab4694ace9bdf0fce06226efa P 64ad8799c8df4b7589869baffb93b68d [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER f8c99fddf5974401b64aa8601605b2b2 (127.16.94.133) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } attrs { replace: true } } peers { permanent_uuid: "f8c99fddf5974401b64aa8601605b2b2" member_type: NON_VOTER last_known_addr { host: "127.16.94.133" port: 42559 } attrs { promote: true } } }
I20240503 14:13:37.864974 20330 raft_consensus.cc:2932] T 220ef8bab4694ace9bdf0fce06226efa P fceaf96557de46c3986c306fa4ac3b9e [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER f8c99fddf5974401b64aa8601605b2b2 (127.16.94.133) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } attrs { replace: true } } peers { permanent_uuid: "f8c99fddf5974401b64aa8601605b2b2" member_type: NON_VOTER last_known_addr { host: "127.16.94.133" port: 42559 } attrs { promote: true } } }
I20240503 14:13:37.865393 20261 raft_consensus.cc:2932] T 220ef8bab4694ace9bdf0fce06226efa P a5d66df07715448fab64928f5bc8299a [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER f8c99fddf5974401b64aa8601605b2b2 (127.16.94.133) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } attrs { replace: true } } peers { permanent_uuid: "f8c99fddf5974401b64aa8601605b2b2" member_type: NON_VOTER last_known_addr { host: "127.16.94.133" port: 42559 } attrs { promote: true } } }
I20240503 14:13:37.870529 20078 catalog_manager.cc:5484] T 220ef8bab4694ace9bdf0fce06226efa P fceaf96557de46c3986c306fa4ac3b9e reported cstate change: config changed from index -1 to 28, NON_VOTER f8c99fddf5974401b64aa8601605b2b2 (127.16.94.133) added. New cstate: current_term: 1 leader_uuid: "64ad8799c8df4b7589869baffb93b68d" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } attrs { replace: true } } peers { permanent_uuid: "f8c99fddf5974401b64aa8601605b2b2" member_type: NON_VOTER last_known_addr { host: "127.16.94.133" port: 42559 } attrs { promote: true } } }
W20240503 14:13:37.872337 20151 consensus_peers.cc:485] T 220ef8bab4694ace9bdf0fce06226efa P 64ad8799c8df4b7589869baffb93b68d -> Peer f8c99fddf5974401b64aa8601605b2b2 (127.16.94.133:42559): Couldn't send request to peer f8c99fddf5974401b64aa8601605b2b2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 220ef8bab4694ace9bdf0fce06226efa. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:37.880347 20261 consensus_queue.cc:237] T 904450ab5e8b4250b7af18b98d2b0985 P a5d66df07715448fab64928f5bc8299a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } attrs { replace: true } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "c4d0f91a34804de99e3a2a255cd06aed" member_type: NON_VOTER last_known_addr { host: "127.16.94.132" port: 41945 } attrs { promote: true } }
I20240503 14:13:37.882825 20179 raft_consensus.cc:1261] T 904450ab5e8b4250b7af18b98d2b0985 P 64ad8799c8df4b7589869baffb93b68d [term 1 FOLLOWER]: Refusing update from remote peer a5d66df07715448fab64928f5bc8299a: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 14:13:37.883481 20847 consensus_queue.cc:1035] T 904450ab5e8b4250b7af18b98d2b0985 P a5d66df07715448fab64928f5bc8299a [LEADER]: Connected to new peer: Peer: permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 14:13:37.885766 20437 raft_consensus.cc:2932] T 904450ab5e8b4250b7af18b98d2b0985 P a5d66df07715448fab64928f5bc8299a [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER c4d0f91a34804de99e3a2a255cd06aed (127.16.94.132) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } attrs { replace: true } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "c4d0f91a34804de99e3a2a255cd06aed" member_type: NON_VOTER last_known_addr { host: "127.16.94.132" port: 41945 } attrs { promote: true } } }
I20240503 14:13:37.886292 20179 raft_consensus.cc:2932] T 904450ab5e8b4250b7af18b98d2b0985 P 64ad8799c8df4b7589869baffb93b68d [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER c4d0f91a34804de99e3a2a255cd06aed (127.16.94.132) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } attrs { replace: true } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "c4d0f91a34804de99e3a2a255cd06aed" member_type: NON_VOTER last_known_addr { host: "127.16.94.132" port: 41945 } attrs { promote: true } } }
I20240503 14:13:37.891287 20077 catalog_manager.cc:5484] T 904450ab5e8b4250b7af18b98d2b0985 P 64ad8799c8df4b7589869baffb93b68d reported cstate change: config changed from index -1 to 28, NON_VOTER c4d0f91a34804de99e3a2a255cd06aed (127.16.94.132) added. New cstate: current_term: 1 leader_uuid: "a5d66df07715448fab64928f5bc8299a" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } attrs { replace: true } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "c4d0f91a34804de99e3a2a255cd06aed" member_type: NON_VOTER last_known_addr { host: "127.16.94.132" port: 41945 } attrs { promote: true } } }
I20240503 14:13:37.905077 20330 raft_consensus.cc:1261] T 904450ab5e8b4250b7af18b98d2b0985 P fceaf96557de46c3986c306fa4ac3b9e [term 1 FOLLOWER]: Refusing update from remote peer a5d66df07715448fab64928f5bc8299a: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 14:13:37.905771 20850 consensus_queue.cc:1035] T 904450ab5e8b4250b7af18b98d2b0985 P a5d66df07715448fab64928f5bc8299a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 14:13:37.907567 20330 raft_consensus.cc:2932] T 904450ab5e8b4250b7af18b98d2b0985 P fceaf96557de46c3986c306fa4ac3b9e [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER c4d0f91a34804de99e3a2a255cd06aed (127.16.94.132) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } attrs { replace: true } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "c4d0f91a34804de99e3a2a255cd06aed" member_type: NON_VOTER last_known_addr { host: "127.16.94.132" port: 41945 } attrs { promote: true } } }
I20240503 14:13:37.925329 20330 consensus_queue.cc:237] T 3054a11a8da4470ea5241b92d164d6a8 P fceaf96557de46c3986c306fa4ac3b9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 2.26, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } attrs { replace: true } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "d6bcbc9fecfb44fa99a81917ea2d559a" member_type: NON_VOTER last_known_addr { host: "127.16.94.134" port: 34983 } attrs { promote: true } }
I20240503 14:13:37.927598 20179 raft_consensus.cc:1261] T 3054a11a8da4470ea5241b92d164d6a8 P 64ad8799c8df4b7589869baffb93b68d [term 2 FOLLOWER]: Refusing update from remote peer fceaf96557de46c3986c306fa4ac3b9e: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240503 14:13:37.928020 20496 consensus_queue.cc:1035] T 3054a11a8da4470ea5241b92d164d6a8 P fceaf96557de46c3986c306fa4ac3b9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
W20240503 14:13:37.929713 20226 consensus_peers.cc:485] T 904450ab5e8b4250b7af18b98d2b0985 P a5d66df07715448fab64928f5bc8299a -> Peer c4d0f91a34804de99e3a2a255cd06aed (127.16.94.132:41945): Couldn't send request to peer c4d0f91a34804de99e3a2a255cd06aed. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 904450ab5e8b4250b7af18b98d2b0985. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:37.930055 20496 raft_consensus.cc:2932] T 3054a11a8da4470ea5241b92d164d6a8 P fceaf96557de46c3986c306fa4ac3b9e [term 2 LEADER]: Committing config change with OpId 2.27: config changed from index -1 to 27, NON_VOTER d6bcbc9fecfb44fa99a81917ea2d559a (127.16.94.134) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } attrs { replace: true } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "d6bcbc9fecfb44fa99a81917ea2d559a" member_type: NON_VOTER last_known_addr { host: "127.16.94.134" port: 34983 } attrs { promote: true } } }
I20240503 14:13:37.930402 20179 raft_consensus.cc:2932] T 3054a11a8da4470ea5241b92d164d6a8 P 64ad8799c8df4b7589869baffb93b68d [term 2 FOLLOWER]: Committing config change with OpId 2.27: config changed from index -1 to 27, NON_VOTER d6bcbc9fecfb44fa99a81917ea2d559a (127.16.94.134) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } attrs { replace: true } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "d6bcbc9fecfb44fa99a81917ea2d559a" member_type: NON_VOTER last_known_addr { host: "127.16.94.134" port: 34983 } attrs { promote: true } } }
I20240503 14:13:37.935420 20179 consensus_queue.cc:237] T 29f4ee03917646f6a5374217bed7a1be P 64ad8799c8df4b7589869baffb93b68d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } attrs { replace: true } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "c4d0f91a34804de99e3a2a255cd06aed" member_type: NON_VOTER last_known_addr { host: "127.16.94.132" port: 41945 } attrs { promote: true } }
I20240503 14:13:37.935350 20077 catalog_manager.cc:5484] T 3054a11a8da4470ea5241b92d164d6a8 P 64ad8799c8df4b7589869baffb93b68d reported cstate change: config changed from index -1 to 27, NON_VOTER d6bcbc9fecfb44fa99a81917ea2d559a (127.16.94.134) added. New cstate: current_term: 2 leader_uuid: "fceaf96557de46c3986c306fa4ac3b9e" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } attrs { replace: true } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "d6bcbc9fecfb44fa99a81917ea2d559a" member_type: NON_VOTER last_known_addr { host: "127.16.94.134" port: 34983 } attrs { promote: true } } }
I20240503 14:13:37.936589 20261 raft_consensus.cc:1261] T 3054a11a8da4470ea5241b92d164d6a8 P a5d66df07715448fab64928f5bc8299a [term 2 FOLLOWER]: Refusing update from remote peer fceaf96557de46c3986c306fa4ac3b9e: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240503 14:13:37.937054 20858 consensus_queue.cc:1035] T 3054a11a8da4470ea5241b92d164d6a8 P fceaf96557de46c3986c306fa4ac3b9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 14:13:37.938342 20261 raft_consensus.cc:2932] T 3054a11a8da4470ea5241b92d164d6a8 P a5d66df07715448fab64928f5bc8299a [term 2 FOLLOWER]: Committing config change with OpId 2.27: config changed from index -1 to 27, NON_VOTER d6bcbc9fecfb44fa99a81917ea2d559a (127.16.94.134) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } attrs { replace: true } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "d6bcbc9fecfb44fa99a81917ea2d559a" member_type: NON_VOTER last_known_addr { host: "127.16.94.134" port: 34983 } attrs { promote: true } } }
I20240503 14:13:37.939404 20259 raft_consensus.cc:1261] T 29f4ee03917646f6a5374217bed7a1be P a5d66df07715448fab64928f5bc8299a [term 1 FOLLOWER]: Refusing update from remote peer 64ad8799c8df4b7589869baffb93b68d: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 14:13:37.939973 20330 raft_consensus.cc:1261] T 29f4ee03917646f6a5374217bed7a1be P fceaf96557de46c3986c306fa4ac3b9e [term 1 FOLLOWER]: Refusing update from remote peer 64ad8799c8df4b7589869baffb93b68d: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 14:13:37.939937 20839 consensus_queue.cc:1035] T 29f4ee03917646f6a5374217bed7a1be P 64ad8799c8df4b7589869baffb93b68d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 14:13:37.940347 20838 consensus_queue.cc:1035] T 29f4ee03917646f6a5374217bed7a1be P 64ad8799c8df4b7589869baffb93b68d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 14:13:37.942679 20838 raft_consensus.cc:2932] T 29f4ee03917646f6a5374217bed7a1be P 64ad8799c8df4b7589869baffb93b68d [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER c4d0f91a34804de99e3a2a255cd06aed (127.16.94.132) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } attrs { replace: true } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "c4d0f91a34804de99e3a2a255cd06aed" member_type: NON_VOTER last_known_addr { host: "127.16.94.132" port: 41945 } attrs { promote: true } } }
I20240503 14:13:37.947183 20330 consensus_queue.cc:237] T 0eb7552695304a1ea6f8e02f189f7295 P fceaf96557de46c3986c306fa4ac3b9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } attrs { replace: true } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "d6bcbc9fecfb44fa99a81917ea2d559a" member_type: NON_VOTER last_known_addr { host: "127.16.94.134" port: 34983 } attrs { promote: true } }
W20240503 14:13:37.947486 20150 consensus_peers.cc:485] T 29f4ee03917646f6a5374217bed7a1be P 64ad8799c8df4b7589869baffb93b68d -> Peer c4d0f91a34804de99e3a2a255cd06aed (127.16.94.132:41945): Couldn't send request to peer c4d0f91a34804de99e3a2a255cd06aed. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 29f4ee03917646f6a5374217bed7a1be. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:37.947458 20329 raft_consensus.cc:2932] T 29f4ee03917646f6a5374217bed7a1be P fceaf96557de46c3986c306fa4ac3b9e [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER c4d0f91a34804de99e3a2a255cd06aed (127.16.94.132) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } attrs { replace: true } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "c4d0f91a34804de99e3a2a255cd06aed" member_type: NON_VOTER last_known_addr { host: "127.16.94.132" port: 41945 } attrs { promote: true } } }
I20240503 14:13:37.948249 20077 catalog_manager.cc:5484] T 29f4ee03917646f6a5374217bed7a1be P 64ad8799c8df4b7589869baffb93b68d reported cstate change: config changed from index -1 to 27, NON_VOTER c4d0f91a34804de99e3a2a255cd06aed (127.16.94.132) added. New cstate: current_term: 1 leader_uuid: "64ad8799c8df4b7589869baffb93b68d" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4d0f91a34804de99e3a2a255cd06aed" member_type: NON_VOTER last_known_addr { host: "127.16.94.132" port: 41945 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:37.950206 20261 raft_consensus.cc:2932] T 29f4ee03917646f6a5374217bed7a1be P a5d66df07715448fab64928f5bc8299a [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER c4d0f91a34804de99e3a2a255cd06aed (127.16.94.132) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } attrs { replace: true } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "c4d0f91a34804de99e3a2a255cd06aed" member_type: NON_VOTER last_known_addr { host: "127.16.94.132" port: 41945 } attrs { promote: true } } }
I20240503 14:13:37.951932 20179 raft_consensus.cc:1261] T 0eb7552695304a1ea6f8e02f189f7295 P 64ad8799c8df4b7589869baffb93b68d [term 1 FOLLOWER]: Refusing update from remote peer fceaf96557de46c3986c306fa4ac3b9e: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 14:13:37.952143 20259 raft_consensus.cc:1261] T 0eb7552695304a1ea6f8e02f189f7295 P a5d66df07715448fab64928f5bc8299a [term 1 FOLLOWER]: Refusing update from remote peer fceaf96557de46c3986c306fa4ac3b9e: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 14:13:37.952487 20496 consensus_queue.cc:1035] T 0eb7552695304a1ea6f8e02f189f7295 P fceaf96557de46c3986c306fa4ac3b9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 14:13:37.952733 20854 consensus_queue.cc:1035] T 0eb7552695304a1ea6f8e02f189f7295 P fceaf96557de46c3986c306fa4ac3b9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 14:13:37.956414 20259 raft_consensus.cc:2932] T 0eb7552695304a1ea6f8e02f189f7295 P a5d66df07715448fab64928f5bc8299a [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER d6bcbc9fecfb44fa99a81917ea2d559a (127.16.94.134) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } attrs { replace: true } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "d6bcbc9fecfb44fa99a81917ea2d559a" member_type: NON_VOTER last_known_addr { host: "127.16.94.134" port: 34983 } attrs { promote: true } } }
I20240503 14:13:37.956606 20496 raft_consensus.cc:2932] T 0eb7552695304a1ea6f8e02f189f7295 P fceaf96557de46c3986c306fa4ac3b9e [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER d6bcbc9fecfb44fa99a81917ea2d559a (127.16.94.134) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } attrs { replace: true } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "d6bcbc9fecfb44fa99a81917ea2d559a" member_type: NON_VOTER last_known_addr { host: "127.16.94.134" port: 34983 } attrs { promote: true } } }
W20240503 14:13:37.958081 20299 consensus_peers.cc:485] T 3054a11a8da4470ea5241b92d164d6a8 P fceaf96557de46c3986c306fa4ac3b9e -> Peer d6bcbc9fecfb44fa99a81917ea2d559a (127.16.94.134:34983): Couldn't send request to peer d6bcbc9fecfb44fa99a81917ea2d559a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3054a11a8da4470ea5241b92d164d6a8. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:37.957875 20180 raft_consensus.cc:2932] T 0eb7552695304a1ea6f8e02f189f7295 P 64ad8799c8df4b7589869baffb93b68d [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER d6bcbc9fecfb44fa99a81917ea2d559a (127.16.94.134) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } attrs { replace: true } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "d6bcbc9fecfb44fa99a81917ea2d559a" member_type: NON_VOTER last_known_addr { host: "127.16.94.134" port: 34983 } attrs { promote: true } } }
W20240503 14:13:37.960695 20299 consensus_peers.cc:485] T 0eb7552695304a1ea6f8e02f189f7295 P fceaf96557de46c3986c306fa4ac3b9e -> Peer d6bcbc9fecfb44fa99a81917ea2d559a (127.16.94.134:34983): Couldn't send request to peer d6bcbc9fecfb44fa99a81917ea2d559a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0eb7552695304a1ea6f8e02f189f7295. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:37.961875 20081 catalog_manager.cc:5484] T 0eb7552695304a1ea6f8e02f189f7295 P 64ad8799c8df4b7589869baffb93b68d reported cstate change: config changed from index -1 to 27, NON_VOTER d6bcbc9fecfb44fa99a81917ea2d559a (127.16.94.134) added. New cstate: current_term: 1 leader_uuid: "fceaf96557de46c3986c306fa4ac3b9e" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } attrs { replace: true } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "d6bcbc9fecfb44fa99a81917ea2d559a" member_type: NON_VOTER last_known_addr { host: "127.16.94.134" port: 34983 } attrs { promote: true } } }
I20240503 14:13:37.964877 20180 consensus_queue.cc:237] T d4743d5a73c54086b277f4c4b1a248fd P 64ad8799c8df4b7589869baffb93b68d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } attrs { replace: true } } peers { permanent_uuid: "f8c99fddf5974401b64aa8601605b2b2" member_type: NON_VOTER last_known_addr { host: "127.16.94.133" port: 42559 } attrs { promote: true } }
W20240503 14:13:37.966871 20151 consensus_peers.cc:485] T d4743d5a73c54086b277f4c4b1a248fd P 64ad8799c8df4b7589869baffb93b68d -> Peer f8c99fddf5974401b64aa8601605b2b2 (127.16.94.133:42559): Couldn't send request to peer f8c99fddf5974401b64aa8601605b2b2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d4743d5a73c54086b277f4c4b1a248fd. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:37.966885 20259 raft_consensus.cc:1261] T d4743d5a73c54086b277f4c4b1a248fd P a5d66df07715448fab64928f5bc8299a [term 2 FOLLOWER]: Refusing update from remote peer 64ad8799c8df4b7589869baffb93b68d: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240503 14:13:37.966885 20329 raft_consensus.cc:1261] T d4743d5a73c54086b277f4c4b1a248fd P fceaf96557de46c3986c306fa4ac3b9e [term 2 FOLLOWER]: Refusing update from remote peer 64ad8799c8df4b7589869baffb93b68d: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240503 14:13:37.967360 20606 consensus_queue.cc:1035] T d4743d5a73c54086b277f4c4b1a248fd P 64ad8799c8df4b7589869baffb93b68d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 14:13:37.967743 20839 consensus_queue.cc:1035] T d4743d5a73c54086b277f4c4b1a248fd P 64ad8799c8df4b7589869baffb93b68d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 14:13:37.969646 20838 raft_consensus.cc:2932] T d4743d5a73c54086b277f4c4b1a248fd P 64ad8799c8df4b7589869baffb93b68d [term 2 LEADER]: Committing config change with OpId 2.28: config changed from index -1 to 28, NON_VOTER f8c99fddf5974401b64aa8601605b2b2 (127.16.94.133) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } attrs { replace: true } } peers { permanent_uuid: "f8c99fddf5974401b64aa8601605b2b2" member_type: NON_VOTER last_known_addr { host: "127.16.94.133" port: 42559 } attrs { promote: true } } }
I20240503 14:13:37.970245 20259 raft_consensus.cc:2932] T d4743d5a73c54086b277f4c4b1a248fd P a5d66df07715448fab64928f5bc8299a [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index -1 to 28, NON_VOTER f8c99fddf5974401b64aa8601605b2b2 (127.16.94.133) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } attrs { replace: true } } peers { permanent_uuid: "f8c99fddf5974401b64aa8601605b2b2" member_type: NON_VOTER last_known_addr { host: "127.16.94.133" port: 42559 } attrs { promote: true } } }
I20240503 14:13:37.970347 20329 raft_consensus.cc:2932] T d4743d5a73c54086b277f4c4b1a248fd P fceaf96557de46c3986c306fa4ac3b9e [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index -1 to 28, NON_VOTER f8c99fddf5974401b64aa8601605b2b2 (127.16.94.133) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } attrs { replace: true } } peers { permanent_uuid: "f8c99fddf5974401b64aa8601605b2b2" member_type: NON_VOTER last_known_addr { host: "127.16.94.133" port: 42559 } attrs { promote: true } } }
I20240503 14:13:37.974388 20077 catalog_manager.cc:5484] T d4743d5a73c54086b277f4c4b1a248fd P 64ad8799c8df4b7589869baffb93b68d reported cstate change: config changed from index -1 to 28, NON_VOTER f8c99fddf5974401b64aa8601605b2b2 (127.16.94.133) added. New cstate: current_term: 2 leader_uuid: "64ad8799c8df4b7589869baffb93b68d" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8c99fddf5974401b64aa8601605b2b2" member_type: NON_VOTER last_known_addr { host: "127.16.94.133" port: 42559 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:38.171216 20683 heartbeater.cc:503] Master 127.16.94.190:39321 was elected leader, sending a full tablet report...
I20240503 14:13:38.218585 20759 heartbeater.cc:503] Master 127.16.94.190:39321 was elected leader, sending a full tablet report...
I20240503 14:13:38.259747 20833 heartbeater.cc:503] Master 127.16.94.190:39321 was elected leader, sending a full tablet report...
I20240503 14:13:38.286998 20136 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:13:38.289121 20136 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:13:38.289299 20136 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:13:38.413461 20860 ts_tablet_manager.cc:902] T 220ef8bab4694ace9bdf0fce06226efa P f8c99fddf5974401b64aa8601605b2b2: Initiating tablet copy from peer 64ad8799c8df4b7589869baffb93b68d (127.16.94.129:37429)
I20240503 14:13:38.414747 20860 tablet_copy_client.cc:308] T 220ef8bab4694ace9bdf0fce06226efa P f8c99fddf5974401b64aa8601605b2b2: tablet copy: Beginning tablet copy session from remote peer at address 127.16.94.129:37429
I20240503 14:13:38.418694 20867 ts_tablet_manager.cc:902] T 29f4ee03917646f6a5374217bed7a1be P c4d0f91a34804de99e3a2a255cd06aed: Initiating tablet copy from peer 64ad8799c8df4b7589869baffb93b68d (127.16.94.129:37429)
I20240503 14:13:38.421617 20867 tablet_copy_client.cc:308] T 29f4ee03917646f6a5374217bed7a1be P c4d0f91a34804de99e3a2a255cd06aed: tablet copy: Beginning tablet copy session from remote peer at address 127.16.94.129:37429
I20240503 14:13:38.426806 20195 tablet_copy_service.cc:140] P 64ad8799c8df4b7589869baffb93b68d: Received BeginTabletCopySession request for tablet 220ef8bab4694ace9bdf0fce06226efa from peer f8c99fddf5974401b64aa8601605b2b2 ({username='slave'} at 127.0.0.1:54550)
I20240503 14:13:38.426959 20195 tablet_copy_service.cc:161] P 64ad8799c8df4b7589869baffb93b68d: Beginning new tablet copy session on tablet 220ef8bab4694ace9bdf0fce06226efa from peer f8c99fddf5974401b64aa8601605b2b2 at {username='slave'} at 127.0.0.1:54550: session id = f8c99fddf5974401b64aa8601605b2b2-220ef8bab4694ace9bdf0fce06226efa
I20240503 14:13:38.429663 20195 tablet_copy_source_session.cc:205] T 220ef8bab4694ace9bdf0fce06226efa P 64ad8799c8df4b7589869baffb93b68d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:38.430912 20860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 220ef8bab4694ace9bdf0fce06226efa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:38.431486 20195 tablet_copy_service.cc:140] P 64ad8799c8df4b7589869baffb93b68d: Received BeginTabletCopySession request for tablet 29f4ee03917646f6a5374217bed7a1be from peer c4d0f91a34804de99e3a2a255cd06aed ({username='slave'} at 127.0.0.1:54566)
I20240503 14:13:38.431589 20195 tablet_copy_service.cc:161] P 64ad8799c8df4b7589869baffb93b68d: Beginning new tablet copy session on tablet 29f4ee03917646f6a5374217bed7a1be from peer c4d0f91a34804de99e3a2a255cd06aed at {username='slave'} at 127.0.0.1:54566: session id = c4d0f91a34804de99e3a2a255cd06aed-29f4ee03917646f6a5374217bed7a1be
I20240503 14:13:38.436969 20860 tablet_copy_client.cc:783] T 220ef8bab4694ace9bdf0fce06226efa P f8c99fddf5974401b64aa8601605b2b2: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:38.437222 20860 tablet_copy_client.cc:647] T 220ef8bab4694ace9bdf0fce06226efa P f8c99fddf5974401b64aa8601605b2b2: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:38.438150 20195 tablet_copy_source_session.cc:205] T 29f4ee03917646f6a5374217bed7a1be P 64ad8799c8df4b7589869baffb93b68d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:38.441480 20874 ts_tablet_manager.cc:902] T 904450ab5e8b4250b7af18b98d2b0985 P c4d0f91a34804de99e3a2a255cd06aed: Initiating tablet copy from peer a5d66df07715448fab64928f5bc8299a (127.16.94.130:39295)
I20240503 14:13:38.442508 20867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29f4ee03917646f6a5374217bed7a1be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:38.443975 20874 tablet_copy_client.cc:308] T 904450ab5e8b4250b7af18b98d2b0985 P c4d0f91a34804de99e3a2a255cd06aed: tablet copy: Beginning tablet copy session from remote peer at address 127.16.94.130:39295
I20240503 14:13:38.446903 20864 tablet_copy_client.cc:1002] T 220ef8bab4694ace9bdf0fce06226efa P f8c99fddf5974401b64aa8601605b2b2: tablet copy: Injecting latency into file download: 2000
I20240503 14:13:38.449018 20867 tablet_copy_client.cc:783] T 29f4ee03917646f6a5374217bed7a1be P c4d0f91a34804de99e3a2a255cd06aed: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:38.449241 20867 tablet_copy_client.cc:647] T 29f4ee03917646f6a5374217bed7a1be P c4d0f91a34804de99e3a2a255cd06aed: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:38.450425 20271 tablet_copy_service.cc:140] P a5d66df07715448fab64928f5bc8299a: Received BeginTabletCopySession request for tablet 904450ab5e8b4250b7af18b98d2b0985 from peer c4d0f91a34804de99e3a2a255cd06aed ({username='slave'} at 127.0.0.1:38712)
I20240503 14:13:38.450562 20271 tablet_copy_service.cc:161] P a5d66df07715448fab64928f5bc8299a: Beginning new tablet copy session on tablet 904450ab5e8b4250b7af18b98d2b0985 from peer c4d0f91a34804de99e3a2a255cd06aed at {username='slave'} at 127.0.0.1:38712: session id = c4d0f91a34804de99e3a2a255cd06aed-904450ab5e8b4250b7af18b98d2b0985
I20240503 14:13:38.450899 20871 tablet_copy_client.cc:1002] T 29f4ee03917646f6a5374217bed7a1be P c4d0f91a34804de99e3a2a255cd06aed: tablet copy: Injecting latency into file download: 2000
I20240503 14:13:38.452801 20271 tablet_copy_source_session.cc:205] T 904450ab5e8b4250b7af18b98d2b0985 P a5d66df07715448fab64928f5bc8299a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:38.453924 20874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 904450ab5e8b4250b7af18b98d2b0985. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:38.459642 20874 tablet_copy_client.cc:783] T 904450ab5e8b4250b7af18b98d2b0985 P c4d0f91a34804de99e3a2a255cd06aed: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:38.459980 20874 tablet_copy_client.cc:647] T 904450ab5e8b4250b7af18b98d2b0985 P c4d0f91a34804de99e3a2a255cd06aed: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:38.461176 20878 tablet_copy_client.cc:1002] T 904450ab5e8b4250b7af18b98d2b0985 P c4d0f91a34804de99e3a2a255cd06aed: tablet copy: Injecting latency into file download: 2000
I20240503 14:13:38.466156 20880 ts_tablet_manager.cc:902] T 0eb7552695304a1ea6f8e02f189f7295 P d6bcbc9fecfb44fa99a81917ea2d559a: Initiating tablet copy from peer fceaf96557de46c3986c306fa4ac3b9e (127.16.94.131:39581)
I20240503 14:13:38.467329 20880 tablet_copy_client.cc:308] T 0eb7552695304a1ea6f8e02f189f7295 P d6bcbc9fecfb44fa99a81917ea2d559a: tablet copy: Beginning tablet copy session from remote peer at address 127.16.94.131:39581
I20240503 14:13:38.473832 20345 tablet_copy_service.cc:140] P fceaf96557de46c3986c306fa4ac3b9e: Received BeginTabletCopySession request for tablet 0eb7552695304a1ea6f8e02f189f7295 from peer d6bcbc9fecfb44fa99a81917ea2d559a ({username='slave'} at 127.0.0.1:56246)
I20240503 14:13:38.473976 20345 tablet_copy_service.cc:161] P fceaf96557de46c3986c306fa4ac3b9e: Beginning new tablet copy session on tablet 0eb7552695304a1ea6f8e02f189f7295 from peer d6bcbc9fecfb44fa99a81917ea2d559a at {username='slave'} at 127.0.0.1:56246: session id = d6bcbc9fecfb44fa99a81917ea2d559a-0eb7552695304a1ea6f8e02f189f7295
I20240503 14:13:38.476029 20345 tablet_copy_source_session.cc:205] T 0eb7552695304a1ea6f8e02f189f7295 P fceaf96557de46c3986c306fa4ac3b9e: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:38.477180 20880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0eb7552695304a1ea6f8e02f189f7295. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:38.482998 20880 tablet_copy_client.cc:783] T 0eb7552695304a1ea6f8e02f189f7295 P d6bcbc9fecfb44fa99a81917ea2d559a: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:38.483261 20880 tablet_copy_client.cc:647] T 0eb7552695304a1ea6f8e02f189f7295 P d6bcbc9fecfb44fa99a81917ea2d559a: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:38.484647 20884 tablet_copy_client.cc:1002] T 0eb7552695304a1ea6f8e02f189f7295 P d6bcbc9fecfb44fa99a81917ea2d559a: tablet copy: Injecting latency into file download: 2000
I20240503 14:13:38.485357 20887 ts_tablet_manager.cc:902] T 3054a11a8da4470ea5241b92d164d6a8 P d6bcbc9fecfb44fa99a81917ea2d559a: Initiating tablet copy from peer fceaf96557de46c3986c306fa4ac3b9e (127.16.94.131:39581)
I20240503 14:13:38.490298 20887 tablet_copy_client.cc:308] T 3054a11a8da4470ea5241b92d164d6a8 P d6bcbc9fecfb44fa99a81917ea2d559a: tablet copy: Beginning tablet copy session from remote peer at address 127.16.94.131:39581
I20240503 14:13:38.490816 20345 tablet_copy_service.cc:140] P fceaf96557de46c3986c306fa4ac3b9e: Received BeginTabletCopySession request for tablet 3054a11a8da4470ea5241b92d164d6a8 from peer d6bcbc9fecfb44fa99a81917ea2d559a ({username='slave'} at 127.0.0.1:56246)
I20240503 14:13:38.490936 20345 tablet_copy_service.cc:161] P fceaf96557de46c3986c306fa4ac3b9e: Beginning new tablet copy session on tablet 3054a11a8da4470ea5241b92d164d6a8 from peer d6bcbc9fecfb44fa99a81917ea2d559a at {username='slave'} at 127.0.0.1:56246: session id = d6bcbc9fecfb44fa99a81917ea2d559a-3054a11a8da4470ea5241b92d164d6a8
I20240503 14:13:38.493404 20345 tablet_copy_source_session.cc:205] T 3054a11a8da4470ea5241b92d164d6a8 P fceaf96557de46c3986c306fa4ac3b9e: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:38.494458 20887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3054a11a8da4470ea5241b92d164d6a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:38.498998 20887 tablet_copy_client.cc:783] T 3054a11a8da4470ea5241b92d164d6a8 P d6bcbc9fecfb44fa99a81917ea2d559a: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:38.499244 20887 tablet_copy_client.cc:647] T 3054a11a8da4470ea5241b92d164d6a8 P d6bcbc9fecfb44fa99a81917ea2d559a: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:38.500461 20891 tablet_copy_client.cc:1002] T 3054a11a8da4470ea5241b92d164d6a8 P d6bcbc9fecfb44fa99a81917ea2d559a: tablet copy: Injecting latency into file download: 2000
I20240503 14:13:38.561550 20892 ts_tablet_manager.cc:902] T d4743d5a73c54086b277f4c4b1a248fd P f8c99fddf5974401b64aa8601605b2b2: Initiating tablet copy from peer 64ad8799c8df4b7589869baffb93b68d (127.16.94.129:37429)
I20240503 14:13:38.562870 20892 tablet_copy_client.cc:308] T d4743d5a73c54086b277f4c4b1a248fd P f8c99fddf5974401b64aa8601605b2b2: tablet copy: Beginning tablet copy session from remote peer at address 127.16.94.129:37429
I20240503 14:13:38.563414 20195 tablet_copy_service.cc:140] P 64ad8799c8df4b7589869baffb93b68d: Received BeginTabletCopySession request for tablet d4743d5a73c54086b277f4c4b1a248fd from peer f8c99fddf5974401b64aa8601605b2b2 ({username='slave'} at 127.0.0.1:54550)
I20240503 14:13:38.563544 20195 tablet_copy_service.cc:161] P 64ad8799c8df4b7589869baffb93b68d: Beginning new tablet copy session on tablet d4743d5a73c54086b277f4c4b1a248fd from peer f8c99fddf5974401b64aa8601605b2b2 at {username='slave'} at 127.0.0.1:54550: session id = f8c99fddf5974401b64aa8601605b2b2-d4743d5a73c54086b277f4c4b1a248fd
I20240503 14:13:38.566367 20195 tablet_copy_source_session.cc:205] T d4743d5a73c54086b277f4c4b1a248fd P 64ad8799c8df4b7589869baffb93b68d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:38.567421 20892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4743d5a73c54086b277f4c4b1a248fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:38.572266 20892 tablet_copy_client.cc:783] T d4743d5a73c54086b277f4c4b1a248fd P f8c99fddf5974401b64aa8601605b2b2: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:38.572544 20892 tablet_copy_client.cc:647] T d4743d5a73c54086b277f4c4b1a248fd P f8c99fddf5974401b64aa8601605b2b2: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:38.574012 20896 tablet_copy_client.cc:1002] T d4743d5a73c54086b277f4c4b1a248fd P f8c99fddf5974401b64aa8601605b2b2: tablet copy: Injecting latency into file download: 2000
I20240503 14:13:40.289589 20136 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:13:40.291904 20136 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:13:40.292074 20136 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:13:40.447280 20860 tablet_copy_client.cc:523] T 220ef8bab4694ace9bdf0fce06226efa P f8c99fddf5974401b64aa8601605b2b2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:40.450789 20860 tablet_bootstrap.cc:492] T 220ef8bab4694ace9bdf0fce06226efa P f8c99fddf5974401b64aa8601605b2b2: Bootstrap starting.
I20240503 14:13:40.451196 20867 tablet_copy_client.cc:523] T 29f4ee03917646f6a5374217bed7a1be P c4d0f91a34804de99e3a2a255cd06aed: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:40.454290 20867 tablet_bootstrap.cc:492] T 29f4ee03917646f6a5374217bed7a1be P c4d0f91a34804de99e3a2a255cd06aed: Bootstrap starting.
I20240503 14:13:40.461536 20874 tablet_copy_client.cc:523] T 904450ab5e8b4250b7af18b98d2b0985 P c4d0f91a34804de99e3a2a255cd06aed: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:40.464778 20874 tablet_bootstrap.cc:492] T 904450ab5e8b4250b7af18b98d2b0985 P c4d0f91a34804de99e3a2a255cd06aed: Bootstrap starting.
I20240503 14:13:40.484951 20880 tablet_copy_client.cc:523] T 0eb7552695304a1ea6f8e02f189f7295 P d6bcbc9fecfb44fa99a81917ea2d559a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:40.488440 20880 tablet_bootstrap.cc:492] T 0eb7552695304a1ea6f8e02f189f7295 P d6bcbc9fecfb44fa99a81917ea2d559a: Bootstrap starting.
I20240503 14:13:40.492846 20867 tablet_bootstrap.cc:492] T 29f4ee03917646f6a5374217bed7a1be P c4d0f91a34804de99e3a2a255cd06aed: Bootstrap replayed 1/1 log segments. Stats: ops{read=27 overwritten=0 applied=27 ignored=0} inserts{seen=103 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:40.493311 20867 tablet_bootstrap.cc:492] T 29f4ee03917646f6a5374217bed7a1be P c4d0f91a34804de99e3a2a255cd06aed: Bootstrap complete.
I20240503 14:13:40.493634 20867 ts_tablet_manager.cc:1367] T 29f4ee03917646f6a5374217bed7a1be P c4d0f91a34804de99e3a2a255cd06aed: Time spent bootstrapping tablet: real 0.039s user 0.030s sys 0.009s
I20240503 14:13:40.494390 20867 raft_consensus.cc:348] T 29f4ee03917646f6a5374217bed7a1be P c4d0f91a34804de99e3a2a255cd06aed [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } attrs { replace: true } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "c4d0f91a34804de99e3a2a255cd06aed" member_type: NON_VOTER last_known_addr { host: "127.16.94.132" port: 41945 } attrs { promote: true } }
I20240503 14:13:40.494753 20867 raft_consensus.cc:729] T 29f4ee03917646f6a5374217bed7a1be P c4d0f91a34804de99e3a2a255cd06aed [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c4d0f91a34804de99e3a2a255cd06aed, State: Initialized, Role: LEARNER
I20240503 14:13:40.495018 20867 consensus_queue.cc:260] T 29f4ee03917646f6a5374217bed7a1be P c4d0f91a34804de99e3a2a255cd06aed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } attrs { replace: true } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "c4d0f91a34804de99e3a2a255cd06aed" member_type: NON_VOTER last_known_addr { host: "127.16.94.132" port: 41945 } attrs { promote: true } }
I20240503 14:13:40.501152 20887 tablet_copy_client.cc:523] T 3054a11a8da4470ea5241b92d164d6a8 P d6bcbc9fecfb44fa99a81917ea2d559a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:40.512588 20887 tablet_bootstrap.cc:492] T 3054a11a8da4470ea5241b92d164d6a8 P d6bcbc9fecfb44fa99a81917ea2d559a: Bootstrap starting.
I20240503 14:13:40.521544 20860 tablet_bootstrap.cc:492] T 220ef8bab4694ace9bdf0fce06226efa P f8c99fddf5974401b64aa8601605b2b2: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=108 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:40.522032 20860 tablet_bootstrap.cc:492] T 220ef8bab4694ace9bdf0fce06226efa P f8c99fddf5974401b64aa8601605b2b2: Bootstrap complete.
I20240503 14:13:40.522372 20860 ts_tablet_manager.cc:1367] T 220ef8bab4694ace9bdf0fce06226efa P f8c99fddf5974401b64aa8601605b2b2: Time spent bootstrapping tablet: real 0.072s user 0.039s sys 0.005s
I20240503 14:13:40.523515 20860 raft_consensus.cc:348] T 220ef8bab4694ace9bdf0fce06226efa P f8c99fddf5974401b64aa8601605b2b2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } attrs { replace: true } } peers { permanent_uuid: "f8c99fddf5974401b64aa8601605b2b2" member_type: NON_VOTER last_known_addr { host: "127.16.94.133" port: 42559 } attrs { promote: true } }
I20240503 14:13:40.523838 20860 raft_consensus.cc:729] T 220ef8bab4694ace9bdf0fce06226efa P f8c99fddf5974401b64aa8601605b2b2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f8c99fddf5974401b64aa8601605b2b2, State: Initialized, Role: LEARNER
I20240503 14:13:40.524080 20860 consensus_queue.cc:260] T 220ef8bab4694ace9bdf0fce06226efa P f8c99fddf5974401b64aa8601605b2b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } attrs { replace: true } } peers { permanent_uuid: "f8c99fddf5974401b64aa8601605b2b2" member_type: NON_VOTER last_known_addr { host: "127.16.94.133" port: 42559 } attrs { promote: true } }
I20240503 14:13:40.533341 20860 ts_tablet_manager.cc:1398] T 220ef8bab4694ace9bdf0fce06226efa P f8c99fddf5974401b64aa8601605b2b2: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20240503 14:13:40.534212 20867 ts_tablet_manager.cc:1398] T 29f4ee03917646f6a5374217bed7a1be P c4d0f91a34804de99e3a2a255cd06aed: Time spent starting tablet: real 0.040s user 0.003s sys 0.032s
I20240503 14:13:40.534919 20195 tablet_copy_service.cc:342] P 64ad8799c8df4b7589869baffb93b68d: Request end of tablet copy session c4d0f91a34804de99e3a2a255cd06aed-29f4ee03917646f6a5374217bed7a1be received from {username='slave'} at 127.0.0.1:54566
I20240503 14:13:40.535027 20195 tablet_copy_service.cc:433] P 64ad8799c8df4b7589869baffb93b68d: ending tablet copy session c4d0f91a34804de99e3a2a255cd06aed-29f4ee03917646f6a5374217bed7a1be on tablet 29f4ee03917646f6a5374217bed7a1be with peer c4d0f91a34804de99e3a2a255cd06aed
I20240503 14:13:40.536430 20194 tablet_copy_service.cc:342] P 64ad8799c8df4b7589869baffb93b68d: Request end of tablet copy session f8c99fddf5974401b64aa8601605b2b2-220ef8bab4694ace9bdf0fce06226efa received from {username='slave'} at 127.0.0.1:54550
I20240503 14:13:40.537274 20194 tablet_copy_service.cc:433] P 64ad8799c8df4b7589869baffb93b68d: ending tablet copy session f8c99fddf5974401b64aa8601605b2b2-220ef8bab4694ace9bdf0fce06226efa on tablet 220ef8bab4694ace9bdf0fce06226efa with peer f8c99fddf5974401b64aa8601605b2b2
I20240503 14:13:40.537688 20658 raft_consensus.cc:1203] T 29f4ee03917646f6a5374217bed7a1be P c4d0f91a34804de99e3a2a255cd06aed [term 1 LEARNER]: Deduplicated request from leader. Original: 1.26->[1.27-1.27] Dedup: 1.27->[]
I20240503 14:13:40.548709 20874 tablet_bootstrap.cc:492] T 904450ab5e8b4250b7af18b98d2b0985 P c4d0f91a34804de99e3a2a255cd06aed: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=111 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:40.549345 20874 tablet_bootstrap.cc:492] T 904450ab5e8b4250b7af18b98d2b0985 P c4d0f91a34804de99e3a2a255cd06aed: Bootstrap complete.
I20240503 14:13:40.549784 20874 ts_tablet_manager.cc:1367] T 904450ab5e8b4250b7af18b98d2b0985 P c4d0f91a34804de99e3a2a255cd06aed: Time spent bootstrapping tablet: real 0.085s user 0.041s sys 0.004s
I20240503 14:13:40.550591 20874 raft_consensus.cc:348] T 904450ab5e8b4250b7af18b98d2b0985 P c4d0f91a34804de99e3a2a255cd06aed [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } attrs { replace: true } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "c4d0f91a34804de99e3a2a255cd06aed" member_type: NON_VOTER last_known_addr { host: "127.16.94.132" port: 41945 } attrs { promote: true } }
I20240503 14:13:40.550900 20874 raft_consensus.cc:729] T 904450ab5e8b4250b7af18b98d2b0985 P c4d0f91a34804de99e3a2a255cd06aed [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c4d0f91a34804de99e3a2a255cd06aed, State: Initialized, Role: LEARNER
I20240503 14:13:40.551131 20874 consensus_queue.cc:260] T 904450ab5e8b4250b7af18b98d2b0985 P c4d0f91a34804de99e3a2a255cd06aed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } attrs { replace: true } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "c4d0f91a34804de99e3a2a255cd06aed" member_type: NON_VOTER last_known_addr { host: "127.16.94.132" port: 41945 } attrs { promote: true } }
I20240503 14:13:40.551806 20874 ts_tablet_manager.cc:1398] T 904450ab5e8b4250b7af18b98d2b0985 P c4d0f91a34804de99e3a2a255cd06aed: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:13:40.553040 20271 tablet_copy_service.cc:342] P a5d66df07715448fab64928f5bc8299a: Request end of tablet copy session c4d0f91a34804de99e3a2a255cd06aed-904450ab5e8b4250b7af18b98d2b0985 received from {username='slave'} at 127.0.0.1:38712
I20240503 14:13:40.553133 20271 tablet_copy_service.cc:433] P a5d66df07715448fab64928f5bc8299a: ending tablet copy session c4d0f91a34804de99e3a2a255cd06aed-904450ab5e8b4250b7af18b98d2b0985 on tablet 904450ab5e8b4250b7af18b98d2b0985 with peer c4d0f91a34804de99e3a2a255cd06aed
I20240503 14:13:40.557637 20880 tablet_bootstrap.cc:492] T 0eb7552695304a1ea6f8e02f189f7295 P d6bcbc9fecfb44fa99a81917ea2d559a: Bootstrap replayed 1/1 log segments. Stats: ops{read=27 overwritten=0 applied=27 ignored=0} inserts{seen=108 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:40.558054 20880 tablet_bootstrap.cc:492] T 0eb7552695304a1ea6f8e02f189f7295 P d6bcbc9fecfb44fa99a81917ea2d559a: Bootstrap complete.
I20240503 14:13:40.558286 20880 ts_tablet_manager.cc:1367] T 0eb7552695304a1ea6f8e02f189f7295 P d6bcbc9fecfb44fa99a81917ea2d559a: Time spent bootstrapping tablet: real 0.070s user 0.049s sys 0.010s
I20240503 14:13:40.558924 20880 raft_consensus.cc:348] T 0eb7552695304a1ea6f8e02f189f7295 P d6bcbc9fecfb44fa99a81917ea2d559a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } attrs { replace: true } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "d6bcbc9fecfb44fa99a81917ea2d559a" member_type: NON_VOTER last_known_addr { host: "127.16.94.134" port: 34983 } attrs { promote: true } }
I20240503 14:13:40.559166 20880 raft_consensus.cc:729] T 0eb7552695304a1ea6f8e02f189f7295 P d6bcbc9fecfb44fa99a81917ea2d559a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d6bcbc9fecfb44fa99a81917ea2d559a, State: Initialized, Role: LEARNER
I20240503 14:13:40.559355 20880 consensus_queue.cc:260] T 0eb7552695304a1ea6f8e02f189f7295 P d6bcbc9fecfb44fa99a81917ea2d559a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } attrs { replace: true } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "d6bcbc9fecfb44fa99a81917ea2d559a" member_type: NON_VOTER last_known_addr { host: "127.16.94.134" port: 34983 } attrs { promote: true } }
I20240503 14:13:40.560189 20880 ts_tablet_manager.cc:1398] T 0eb7552695304a1ea6f8e02f189f7295 P d6bcbc9fecfb44fa99a81917ea2d559a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:40.560804 20345 tablet_copy_service.cc:342] P fceaf96557de46c3986c306fa4ac3b9e: Request end of tablet copy session d6bcbc9fecfb44fa99a81917ea2d559a-0eb7552695304a1ea6f8e02f189f7295 received from {username='slave'} at 127.0.0.1:56246
I20240503 14:13:40.560902 20345 tablet_copy_service.cc:433] P fceaf96557de46c3986c306fa4ac3b9e: ending tablet copy session d6bcbc9fecfb44fa99a81917ea2d559a-0eb7552695304a1ea6f8e02f189f7295 on tablet 0eb7552695304a1ea6f8e02f189f7295 with peer d6bcbc9fecfb44fa99a81917ea2d559a
I20240503 14:13:40.574337 20892 tablet_copy_client.cc:523] T d4743d5a73c54086b277f4c4b1a248fd P f8c99fddf5974401b64aa8601605b2b2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:40.575006 20887 tablet_bootstrap.cc:492] T 3054a11a8da4470ea5241b92d164d6a8 P d6bcbc9fecfb44fa99a81917ea2d559a: Bootstrap replayed 1/1 log segments. Stats: ops{read=27 overwritten=0 applied=27 ignored=0} inserts{seen=94 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:40.575464 20887 tablet_bootstrap.cc:492] T 3054a11a8da4470ea5241b92d164d6a8 P d6bcbc9fecfb44fa99a81917ea2d559a: Bootstrap complete.
I20240503 14:13:40.575728 20887 ts_tablet_manager.cc:1367] T 3054a11a8da4470ea5241b92d164d6a8 P d6bcbc9fecfb44fa99a81917ea2d559a: Time spent bootstrapping tablet: real 0.063s user 0.039s sys 0.002s
I20240503 14:13:40.576475 20887 raft_consensus.cc:348] T 3054a11a8da4470ea5241b92d164d6a8 P d6bcbc9fecfb44fa99a81917ea2d559a [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } attrs { replace: true } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "d6bcbc9fecfb44fa99a81917ea2d559a" member_type: NON_VOTER last_known_addr { host: "127.16.94.134" port: 34983 } attrs { promote: true } }
I20240503 14:13:40.576700 20887 raft_consensus.cc:729] T 3054a11a8da4470ea5241b92d164d6a8 P d6bcbc9fecfb44fa99a81917ea2d559a [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: d6bcbc9fecfb44fa99a81917ea2d559a, State: Initialized, Role: LEARNER
I20240503 14:13:40.576915 20887 consensus_queue.cc:260] T 3054a11a8da4470ea5241b92d164d6a8 P d6bcbc9fecfb44fa99a81917ea2d559a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 27, Last appended: 2.27, Last appended by leader: 27, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } attrs { replace: true } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "d6bcbc9fecfb44fa99a81917ea2d559a" member_type: NON_VOTER last_known_addr { host: "127.16.94.134" port: 34983 } attrs { promote: true } }
I20240503 14:13:40.577499 20887 ts_tablet_manager.cc:1398] T 3054a11a8da4470ea5241b92d164d6a8 P d6bcbc9fecfb44fa99a81917ea2d559a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:13:40.577857 20892 tablet_bootstrap.cc:492] T d4743d5a73c54086b277f4c4b1a248fd P f8c99fddf5974401b64aa8601605b2b2: Bootstrap starting.
I20240503 14:13:40.579018 20345 tablet_copy_service.cc:342] P fceaf96557de46c3986c306fa4ac3b9e: Request end of tablet copy session d6bcbc9fecfb44fa99a81917ea2d559a-3054a11a8da4470ea5241b92d164d6a8 received from {username='slave'} at 127.0.0.1:56246
I20240503 14:13:40.579134 20345 tablet_copy_service.cc:433] P fceaf96557de46c3986c306fa4ac3b9e: ending tablet copy session d6bcbc9fecfb44fa99a81917ea2d559a-3054a11a8da4470ea5241b92d164d6a8 on tablet 3054a11a8da4470ea5241b92d164d6a8 with peer d6bcbc9fecfb44fa99a81917ea2d559a
I20240503 14:13:40.579897 20807 raft_consensus.cc:1203] T 3054a11a8da4470ea5241b92d164d6a8 P d6bcbc9fecfb44fa99a81917ea2d559a [term 2 LEARNER]: Deduplicated request from leader. Original: 2.26->[2.27-2.27] Dedup: 2.27->[]
I20240503 14:13:40.611392 20892 tablet_bootstrap.cc:492] T d4743d5a73c54086b277f4c4b1a248fd P f8c99fddf5974401b64aa8601605b2b2: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=129 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:40.611728 20892 tablet_bootstrap.cc:492] T d4743d5a73c54086b277f4c4b1a248fd P f8c99fddf5974401b64aa8601605b2b2: Bootstrap complete.
I20240503 14:13:40.611944 20892 ts_tablet_manager.cc:1367] T d4743d5a73c54086b277f4c4b1a248fd P f8c99fddf5974401b64aa8601605b2b2: Time spent bootstrapping tablet: real 0.034s user 0.031s sys 0.000s
I20240503 14:13:40.612471 20892 raft_consensus.cc:348] T d4743d5a73c54086b277f4c4b1a248fd P f8c99fddf5974401b64aa8601605b2b2 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } attrs { replace: true } } peers { permanent_uuid: "f8c99fddf5974401b64aa8601605b2b2" member_type: NON_VOTER last_known_addr { host: "127.16.94.133" port: 42559 } attrs { promote: true } }
I20240503 14:13:40.612648 20892 raft_consensus.cc:729] T d4743d5a73c54086b277f4c4b1a248fd P f8c99fddf5974401b64aa8601605b2b2 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: f8c99fddf5974401b64aa8601605b2b2, State: Initialized, Role: LEARNER
I20240503 14:13:40.612888 20892 consensus_queue.cc:260] T d4743d5a73c54086b277f4c4b1a248fd P f8c99fddf5974401b64aa8601605b2b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 2.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } attrs { replace: true } } peers { permanent_uuid: "f8c99fddf5974401b64aa8601605b2b2" member_type: NON_VOTER last_known_addr { host: "127.16.94.133" port: 42559 } attrs { promote: true } }
I20240503 14:13:40.613394 20892 ts_tablet_manager.cc:1398] T d4743d5a73c54086b277f4c4b1a248fd P f8c99fddf5974401b64aa8601605b2b2: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240503 14:13:40.614015 20194 tablet_copy_service.cc:342] P 64ad8799c8df4b7589869baffb93b68d: Request end of tablet copy session f8c99fddf5974401b64aa8601605b2b2-d4743d5a73c54086b277f4c4b1a248fd received from {username='slave'} at 127.0.0.1:54550
I20240503 14:13:40.614110 20194 tablet_copy_service.cc:433] P 64ad8799c8df4b7589869baffb93b68d: ending tablet copy session f8c99fddf5974401b64aa8601605b2b2-d4743d5a73c54086b277f4c4b1a248fd on tablet d4743d5a73c54086b277f4c4b1a248fd with peer f8c99fddf5974401b64aa8601605b2b2
I20240503 14:13:40.683468 20658 raft_consensus.cc:1203] T 904450ab5e8b4250b7af18b98d2b0985 P c4d0f91a34804de99e3a2a255cd06aed [term 1 LEARNER]: Deduplicated request from leader. Original: 1.27->[1.28-1.28] Dedup: 1.28->[]
I20240503 14:13:40.718941 20734 raft_consensus.cc:1203] T d4743d5a73c54086b277f4c4b1a248fd P f8c99fddf5974401b64aa8601605b2b2 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.27->[2.28-2.28] Dedup: 2.28->[]
I20240503 14:13:40.889992 20807 raft_consensus.cc:1203] T 0eb7552695304a1ea6f8e02f189f7295 P d6bcbc9fecfb44fa99a81917ea2d559a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.26->[1.27-1.27] Dedup: 1.27->[]
I20240503 14:13:40.946322 20734 raft_consensus.cc:1203] T 220ef8bab4694ace9bdf0fce06226efa P f8c99fddf5974401b64aa8601605b2b2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.27->[1.28-1.28] Dedup: 1.28->[]
I20240503 14:13:41.074179 20919 raft_consensus.cc:1050] T 29f4ee03917646f6a5374217bed7a1be P 64ad8799c8df4b7589869baffb93b68d: attempting to promote NON_VOTER c4d0f91a34804de99e3a2a255cd06aed to VOTER
I20240503 14:13:41.074904 20919 consensus_queue.cc:237] T 29f4ee03917646f6a5374217bed7a1be P 64ad8799c8df4b7589869baffb93b68d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } attrs { replace: true } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "c4d0f91a34804de99e3a2a255cd06aed" member_type: VOTER last_known_addr { host: "127.16.94.132" port: 41945 } attrs { promote: false } }
I20240503 14:13:41.077081 20329 raft_consensus.cc:1261] T 29f4ee03917646f6a5374217bed7a1be P fceaf96557de46c3986c306fa4ac3b9e [term 1 FOLLOWER]: Refusing update from remote peer 64ad8799c8df4b7589869baffb93b68d: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 14:13:41.077090 20658 raft_consensus.cc:1261] T 29f4ee03917646f6a5374217bed7a1be P c4d0f91a34804de99e3a2a255cd06aed [term 1 LEARNER]: Refusing update from remote peer 64ad8799c8df4b7589869baffb93b68d: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 14:13:41.077517 20259 raft_consensus.cc:1261] T 29f4ee03917646f6a5374217bed7a1be P a5d66df07715448fab64928f5bc8299a [term 1 FOLLOWER]: Refusing update from remote peer 64ad8799c8df4b7589869baffb93b68d: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 14:13:41.077790 20919 consensus_queue.cc:1035] T 29f4ee03917646f6a5374217bed7a1be P 64ad8799c8df4b7589869baffb93b68d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 14:13:41.078150 20918 consensus_queue.cc:1035] T 29f4ee03917646f6a5374217bed7a1be P 64ad8799c8df4b7589869baffb93b68d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4d0f91a34804de99e3a2a255cd06aed" member_type: VOTER last_known_addr { host: "127.16.94.132" port: 41945 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 14:13:41.078428 20901 consensus_queue.cc:1035] T 29f4ee03917646f6a5374217bed7a1be P 64ad8799c8df4b7589869baffb93b68d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 14:13:41.081907 20918 raft_consensus.cc:2932] T 29f4ee03917646f6a5374217bed7a1be P 64ad8799c8df4b7589869baffb93b68d [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index 27 to 28, c4d0f91a34804de99e3a2a255cd06aed (127.16.94.132) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } attrs { replace: true } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "c4d0f91a34804de99e3a2a255cd06aed" member_type: VOTER last_known_addr { host: "127.16.94.132" port: 41945 } attrs { promote: false } } }
I20240503 14:13:41.082437 20658 raft_consensus.cc:2932] T 29f4ee03917646f6a5374217bed7a1be P c4d0f91a34804de99e3a2a255cd06aed [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, c4d0f91a34804de99e3a2a255cd06aed (127.16.94.132) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } attrs { replace: true } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "c4d0f91a34804de99e3a2a255cd06aed" member_type: VOTER last_known_addr { host: "127.16.94.132" port: 41945 } attrs { promote: false } } }
I20240503 14:13:41.084436 20261 raft_consensus.cc:2932] T 29f4ee03917646f6a5374217bed7a1be P a5d66df07715448fab64928f5bc8299a [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, c4d0f91a34804de99e3a2a255cd06aed (127.16.94.132) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } attrs { replace: true } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "c4d0f91a34804de99e3a2a255cd06aed" member_type: VOTER last_known_addr { host: "127.16.94.132" port: 41945 } attrs { promote: false } } }
I20240503 14:13:41.086500 20329 raft_consensus.cc:2932] T 29f4ee03917646f6a5374217bed7a1be P fceaf96557de46c3986c306fa4ac3b9e [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, c4d0f91a34804de99e3a2a255cd06aed (127.16.94.132) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } attrs { replace: true } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "c4d0f91a34804de99e3a2a255cd06aed" member_type: VOTER last_known_addr { host: "127.16.94.132" port: 41945 } attrs { promote: false } } }
I20240503 14:13:41.088217 20078 catalog_manager.cc:5484] T 29f4ee03917646f6a5374217bed7a1be P 64ad8799c8df4b7589869baffb93b68d reported cstate change: config changed from index 27 to 28, c4d0f91a34804de99e3a2a255cd06aed (127.16.94.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "64ad8799c8df4b7589869baffb93b68d" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4d0f91a34804de99e3a2a255cd06aed" member_type: VOTER last_known_addr { host: "127.16.94.132" port: 41945 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:41.096279 20180 consensus_queue.cc:237] T 29f4ee03917646f6a5374217bed7a1be P 64ad8799c8df4b7589869baffb93b68d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "c4d0f91a34804de99e3a2a255cd06aed" member_type: VOTER last_known_addr { host: "127.16.94.132" port: 41945 } attrs { promote: false } }
I20240503 14:13:41.097762 20330 raft_consensus.cc:1261] T 29f4ee03917646f6a5374217bed7a1be P fceaf96557de46c3986c306fa4ac3b9e [term 1 FOLLOWER]: Refusing update from remote peer 64ad8799c8df4b7589869baffb93b68d: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240503 14:13:41.097823 20658 raft_consensus.cc:1261] T 29f4ee03917646f6a5374217bed7a1be P c4d0f91a34804de99e3a2a255cd06aed [term 1 FOLLOWER]: Refusing update from remote peer 64ad8799c8df4b7589869baffb93b68d: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240503 14:13:41.098240 20921 consensus_queue.cc:1035] T 29f4ee03917646f6a5374217bed7a1be P 64ad8799c8df4b7589869baffb93b68d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4d0f91a34804de99e3a2a255cd06aed" member_type: VOTER last_known_addr { host: "127.16.94.132" port: 41945 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 14:13:41.098570 20919 consensus_queue.cc:1035] T 29f4ee03917646f6a5374217bed7a1be P 64ad8799c8df4b7589869baffb93b68d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 14:13:41.100597 20921 raft_consensus.cc:2932] T 29f4ee03917646f6a5374217bed7a1be P 64ad8799c8df4b7589869baffb93b68d [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER a5d66df07715448fab64928f5bc8299a (127.16.94.130) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "c4d0f91a34804de99e3a2a255cd06aed" member_type: VOTER last_known_addr { host: "127.16.94.132" port: 41945 } attrs { promote: false } } }
I20240503 14:13:41.101161 20658 raft_consensus.cc:2932] T 29f4ee03917646f6a5374217bed7a1be P c4d0f91a34804de99e3a2a255cd06aed [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER a5d66df07715448fab64928f5bc8299a (127.16.94.130) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "c4d0f91a34804de99e3a2a255cd06aed" member_type: VOTER last_known_addr { host: "127.16.94.132" port: 41945 } attrs { promote: false } } }
I20240503 14:13:41.102285 20330 raft_consensus.cc:2932] T 29f4ee03917646f6a5374217bed7a1be P fceaf96557de46c3986c306fa4ac3b9e [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER a5d66df07715448fab64928f5bc8299a (127.16.94.130) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "c4d0f91a34804de99e3a2a255cd06aed" member_type: VOTER last_known_addr { host: "127.16.94.132" port: 41945 } attrs { promote: false } } }
I20240503 14:13:41.106006 20080 catalog_manager.cc:5484] T 29f4ee03917646f6a5374217bed7a1be P c4d0f91a34804de99e3a2a255cd06aed reported cstate change: config changed from index 28 to 29, VOTER a5d66df07715448fab64928f5bc8299a (127.16.94.130) evicted. New cstate: current_term: 1 leader_uuid: "64ad8799c8df4b7589869baffb93b68d" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "c4d0f91a34804de99e3a2a255cd06aed" member_type: VOTER last_known_addr { host: "127.16.94.132" port: 41945 } attrs { promote: false } } }
I20240503 14:13:41.108050 20066 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 29f4ee03917646f6a5374217bed7a1be with cas_config_opid_index 28: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:41.109112 20249 tablet_service.cc:1514] Processing DeleteTablet for tablet 29f4ee03917646f6a5374217bed7a1be with delete_type TABLET_DATA_TOMBSTONED (TS a5d66df07715448fab64928f5bc8299a not found in new config with opid_index 29) from {username='slave'} at 127.0.0.1:38642
I20240503 14:13:41.110098 20929 tablet_replica.cc:330] T 29f4ee03917646f6a5374217bed7a1be P a5d66df07715448fab64928f5bc8299a: stopping tablet replica
I20240503 14:13:41.110278 20929 raft_consensus.cc:2229] T 29f4ee03917646f6a5374217bed7a1be P a5d66df07715448fab64928f5bc8299a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:41.110692 20929 raft_consensus.cc:2258] T 29f4ee03917646f6a5374217bed7a1be P a5d66df07715448fab64928f5bc8299a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:41.111605 20929 ts_tablet_manager.cc:1875] T 29f4ee03917646f6a5374217bed7a1be P a5d66df07715448fab64928f5bc8299a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:41.118633 20929 ts_tablet_manager.cc:1888] T 29f4ee03917646f6a5374217bed7a1be P a5d66df07715448fab64928f5bc8299a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.28
I20240503 14:13:41.118842 20929 log.cc:1196] T 29f4ee03917646f6a5374217bed7a1be P a5d66df07715448fab64928f5bc8299a: Deleting WAL directory at /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714745515241994-16762-0/minicluster-data/ts-1-root/wals/29f4ee03917646f6a5374217bed7a1be
I20240503 14:13:41.119603 20069 catalog_manager.cc:4830] TS a5d66df07715448fab64928f5bc8299a (127.16.94.130:39295): tablet 29f4ee03917646f6a5374217bed7a1be (table test-workload [id=cc8b29b2b6ee4a4eb4823287254752f5]) successfully deleted
I20240503 14:13:41.128145 20931 raft_consensus.cc:1050] T 904450ab5e8b4250b7af18b98d2b0985 P a5d66df07715448fab64928f5bc8299a: attempting to promote NON_VOTER c4d0f91a34804de99e3a2a255cd06aed to VOTER
I20240503 14:13:41.128783 20931 consensus_queue.cc:237] T 904450ab5e8b4250b7af18b98d2b0985 P a5d66df07715448fab64928f5bc8299a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } attrs { replace: true } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "c4d0f91a34804de99e3a2a255cd06aed" member_type: VOTER last_known_addr { host: "127.16.94.132" port: 41945 } attrs { promote: false } }
I20240503 14:13:41.131160 20658 raft_consensus.cc:1261] T 904450ab5e8b4250b7af18b98d2b0985 P c4d0f91a34804de99e3a2a255cd06aed [term 1 LEARNER]: Refusing update from remote peer a5d66df07715448fab64928f5bc8299a: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240503 14:13:41.131558 20329 raft_consensus.cc:1261] T 904450ab5e8b4250b7af18b98d2b0985 P fceaf96557de46c3986c306fa4ac3b9e [term 1 FOLLOWER]: Refusing update from remote peer a5d66df07715448fab64928f5bc8299a: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240503 14:13:41.131558 20179 raft_consensus.cc:1261] T 904450ab5e8b4250b7af18b98d2b0985 P 64ad8799c8df4b7589869baffb93b68d [term 1 FOLLOWER]: Refusing update from remote peer a5d66df07715448fab64928f5bc8299a: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240503 14:13:41.131765 20930 consensus_queue.cc:1035] T 904450ab5e8b4250b7af18b98d2b0985 P a5d66df07715448fab64928f5bc8299a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4d0f91a34804de99e3a2a255cd06aed" member_type: VOTER last_known_addr { host: "127.16.94.132" port: 41945 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 14:13:41.132086 20931 consensus_queue.cc:1035] T 904450ab5e8b4250b7af18b98d2b0985 P a5d66df07715448fab64928f5bc8299a [LEADER]: Connected to new peer: Peer: permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 14:13:41.132341 20905 consensus_queue.cc:1035] T 904450ab5e8b4250b7af18b98d2b0985 P a5d66df07715448fab64928f5bc8299a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 14:13:41.135483 20931 raft_consensus.cc:2932] T 904450ab5e8b4250b7af18b98d2b0985 P a5d66df07715448fab64928f5bc8299a [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, c4d0f91a34804de99e3a2a255cd06aed (127.16.94.132) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } attrs { replace: true } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "c4d0f91a34804de99e3a2a255cd06aed" member_type: VOTER last_known_addr { host: "127.16.94.132" port: 41945 } attrs { promote: false } } }
I20240503 14:13:41.136117 20658 raft_consensus.cc:2932] T 904450ab5e8b4250b7af18b98d2b0985 P c4d0f91a34804de99e3a2a255cd06aed [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, c4d0f91a34804de99e3a2a255cd06aed (127.16.94.132) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } attrs { replace: true } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "c4d0f91a34804de99e3a2a255cd06aed" member_type: VOTER last_known_addr { host: "127.16.94.132" port: 41945 } attrs { promote: false } } }
I20240503 14:13:41.138921 20179 raft_consensus.cc:2932] T 904450ab5e8b4250b7af18b98d2b0985 P 64ad8799c8df4b7589869baffb93b68d [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, c4d0f91a34804de99e3a2a255cd06aed (127.16.94.132) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } attrs { replace: true } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "c4d0f91a34804de99e3a2a255cd06aed" member_type: VOTER last_known_addr { host: "127.16.94.132" port: 41945 } attrs { promote: false } } }
I20240503 14:13:41.142273 20077 catalog_manager.cc:5484] T 904450ab5e8b4250b7af18b98d2b0985 P a5d66df07715448fab64928f5bc8299a reported cstate change: config changed from index 28 to 29, c4d0f91a34804de99e3a2a255cd06aed (127.16.94.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a5d66df07715448fab64928f5bc8299a" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4d0f91a34804de99e3a2a255cd06aed" member_type: VOTER last_known_addr { host: "127.16.94.132" port: 41945 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:41.147141 20329 raft_consensus.cc:2932] T 904450ab5e8b4250b7af18b98d2b0985 P fceaf96557de46c3986c306fa4ac3b9e [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, c4d0f91a34804de99e3a2a255cd06aed (127.16.94.132) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } attrs { replace: true } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "c4d0f91a34804de99e3a2a255cd06aed" member_type: VOTER last_known_addr { host: "127.16.94.132" port: 41945 } attrs { promote: false } } }
I20240503 14:13:41.149571 20261 consensus_queue.cc:237] T 904450ab5e8b4250b7af18b98d2b0985 P a5d66df07715448fab64928f5bc8299a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "c4d0f91a34804de99e3a2a255cd06aed" member_type: VOTER last_known_addr { host: "127.16.94.132" port: 41945 } attrs { promote: false } }
I20240503 14:13:41.151178 20658 raft_consensus.cc:1261] T 904450ab5e8b4250b7af18b98d2b0985 P c4d0f91a34804de99e3a2a255cd06aed [term 1 FOLLOWER]: Refusing update from remote peer a5d66df07715448fab64928f5bc8299a: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240503 14:13:41.151587 20932 consensus_queue.cc:1035] T 904450ab5e8b4250b7af18b98d2b0985 P a5d66df07715448fab64928f5bc8299a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4d0f91a34804de99e3a2a255cd06aed" member_type: VOTER last_known_addr { host: "127.16.94.132" port: 41945 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 14:13:41.151858 20330 raft_consensus.cc:1261] T 904450ab5e8b4250b7af18b98d2b0985 P fceaf96557de46c3986c306fa4ac3b9e [term 1 FOLLOWER]: Refusing update from remote peer a5d66df07715448fab64928f5bc8299a: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240503 14:13:41.152292 20932 consensus_queue.cc:1035] T 904450ab5e8b4250b7af18b98d2b0985 P a5d66df07715448fab64928f5bc8299a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 14:13:41.154389 20901 raft_consensus.cc:1050] T d4743d5a73c54086b277f4c4b1a248fd P 64ad8799c8df4b7589869baffb93b68d: attempting to promote NON_VOTER f8c99fddf5974401b64aa8601605b2b2 to VOTER
I20240503 14:13:41.154764 20905 raft_consensus.cc:2932] T 904450ab5e8b4250b7af18b98d2b0985 P a5d66df07715448fab64928f5bc8299a [term 1 LEADER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER 64ad8799c8df4b7589869baffb93b68d (127.16.94.129) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "c4d0f91a34804de99e3a2a255cd06aed" member_type: VOTER last_known_addr { host: "127.16.94.132" port: 41945 } attrs { promote: false } } }
I20240503 14:13:41.155162 20901 consensus_queue.cc:237] T d4743d5a73c54086b277f4c4b1a248fd P 64ad8799c8df4b7589869baffb93b68d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } attrs { replace: true } } peers { permanent_uuid: "f8c99fddf5974401b64aa8601605b2b2" member_type: VOTER last_known_addr { host: "127.16.94.133" port: 42559 } attrs { promote: false } }
I20240503 14:13:41.157058 20330 raft_consensus.cc:2932] T 904450ab5e8b4250b7af18b98d2b0985 P fceaf96557de46c3986c306fa4ac3b9e [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER 64ad8799c8df4b7589869baffb93b68d (127.16.94.129) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "c4d0f91a34804de99e3a2a255cd06aed" member_type: VOTER last_known_addr { host: "127.16.94.132" port: 41945 } attrs { promote: false } } }
I20240503 14:13:41.157330 20658 raft_consensus.cc:2932] T 904450ab5e8b4250b7af18b98d2b0985 P c4d0f91a34804de99e3a2a255cd06aed [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER 64ad8799c8df4b7589869baffb93b68d (127.16.94.129) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "c4d0f91a34804de99e3a2a255cd06aed" member_type: VOTER last_known_addr { host: "127.16.94.132" port: 41945 } attrs { promote: false } } }
I20240503 14:13:41.158219 20329 raft_consensus.cc:1261] T d4743d5a73c54086b277f4c4b1a248fd P fceaf96557de46c3986c306fa4ac3b9e [term 2 FOLLOWER]: Refusing update from remote peer 64ad8799c8df4b7589869baffb93b68d: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240503 14:13:41.159441 20069 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 904450ab5e8b4250b7af18b98d2b0985 with cas_config_opid_index 29: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:41.159674 20918 consensus_queue.cc:1035] T d4743d5a73c54086b277f4c4b1a248fd P 64ad8799c8df4b7589869baffb93b68d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 14:13:41.160955 20734 raft_consensus.cc:1261] T d4743d5a73c54086b277f4c4b1a248fd P f8c99fddf5974401b64aa8601605b2b2 [term 2 LEARNER]: Refusing update from remote peer 64ad8799c8df4b7589869baffb93b68d: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240503 14:13:41.161329 20261 raft_consensus.cc:1261] T d4743d5a73c54086b277f4c4b1a248fd P a5d66df07715448fab64928f5bc8299a [term 2 FOLLOWER]: Refusing update from remote peer 64ad8799c8df4b7589869baffb93b68d: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240503 14:13:41.161243 20081 catalog_manager.cc:5484] T 904450ab5e8b4250b7af18b98d2b0985 P a5d66df07715448fab64928f5bc8299a reported cstate change: config changed from index 29 to 30, VOTER 64ad8799c8df4b7589869baffb93b68d (127.16.94.129) evicted. New cstate: current_term: 1 leader_uuid: "a5d66df07715448fab64928f5bc8299a" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4d0f91a34804de99e3a2a255cd06aed" member_type: VOTER last_known_addr { host: "127.16.94.132" port: 41945 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:41.162554 20921 consensus_queue.cc:1035] T d4743d5a73c54086b277f4c4b1a248fd P 64ad8799c8df4b7589869baffb93b68d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c99fddf5974401b64aa8601605b2b2" member_type: VOTER last_known_addr { host: "127.16.94.133" port: 42559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 14:13:41.162784 20918 consensus_queue.cc:1035] T d4743d5a73c54086b277f4c4b1a248fd P 64ad8799c8df4b7589869baffb93b68d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 14:13:41.165467 20901 raft_consensus.cc:2932] T d4743d5a73c54086b277f4c4b1a248fd P 64ad8799c8df4b7589869baffb93b68d [term 2 LEADER]: Committing config change with OpId 2.29: config changed from index 28 to 29, f8c99fddf5974401b64aa8601605b2b2 (127.16.94.133) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } attrs { replace: true } } peers { permanent_uuid: "f8c99fddf5974401b64aa8601605b2b2" member_type: VOTER last_known_addr { host: "127.16.94.133" port: 42559 } attrs { promote: false } } }
I20240503 14:13:41.168787 20172 tablet_service.cc:1514] Processing DeleteTablet for tablet 904450ab5e8b4250b7af18b98d2b0985 with delete_type TABLET_DATA_TOMBSTONED (TS 64ad8799c8df4b7589869baffb93b68d not found in new config with opid_index 30) from {username='slave'} at 127.0.0.1:54496
I20240503 14:13:41.168922 20734 raft_consensus.cc:2932] T d4743d5a73c54086b277f4c4b1a248fd P f8c99fddf5974401b64aa8601605b2b2 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 28 to 29, f8c99fddf5974401b64aa8601605b2b2 (127.16.94.133) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } attrs { replace: true } } peers { permanent_uuid: "f8c99fddf5974401b64aa8601605b2b2" member_type: VOTER last_known_addr { host: "127.16.94.133" port: 42559 } attrs { promote: false } } }
I20240503 14:13:41.169427 20329 raft_consensus.cc:2932] T d4743d5a73c54086b277f4c4b1a248fd P fceaf96557de46c3986c306fa4ac3b9e [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 28 to 29, f8c99fddf5974401b64aa8601605b2b2 (127.16.94.133) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } attrs { replace: true } } peers { permanent_uuid: "f8c99fddf5974401b64aa8601605b2b2" member_type: VOTER last_known_addr { host: "127.16.94.133" port: 42559 } attrs { promote: false } } }
I20240503 14:13:41.169875 20261 raft_consensus.cc:2932] T d4743d5a73c54086b277f4c4b1a248fd P a5d66df07715448fab64928f5bc8299a [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 28 to 29, f8c99fddf5974401b64aa8601605b2b2 (127.16.94.133) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } attrs { replace: true } } peers { permanent_uuid: "f8c99fddf5974401b64aa8601605b2b2" member_type: VOTER last_known_addr { host: "127.16.94.133" port: 42559 } attrs { promote: false } } }
I20240503 14:13:41.172997 20941 tablet_replica.cc:330] T 904450ab5e8b4250b7af18b98d2b0985 P 64ad8799c8df4b7589869baffb93b68d: stopping tablet replica
I20240503 14:13:41.173166 20941 raft_consensus.cc:2229] T 904450ab5e8b4250b7af18b98d2b0985 P 64ad8799c8df4b7589869baffb93b68d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:41.173575 20941 raft_consensus.cc:2258] T 904450ab5e8b4250b7af18b98d2b0985 P 64ad8799c8df4b7589869baffb93b68d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:41.173564 20080 catalog_manager.cc:5484] T d4743d5a73c54086b277f4c4b1a248fd P fceaf96557de46c3986c306fa4ac3b9e reported cstate change: config changed from index 28 to 29, f8c99fddf5974401b64aa8601605b2b2 (127.16.94.133) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "64ad8799c8df4b7589869baffb93b68d" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } attrs { replace: true } } peers { permanent_uuid: "f8c99fddf5974401b64aa8601605b2b2" member_type: VOTER last_known_addr { host: "127.16.94.133" port: 42559 } attrs { promote: false } } }
I20240503 14:13:41.176816 20941 ts_tablet_manager.cc:1875] T 904450ab5e8b4250b7af18b98d2b0985 P 64ad8799c8df4b7589869baffb93b68d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:41.179332 20180 consensus_queue.cc:237] T d4743d5a73c54086b277f4c4b1a248fd P 64ad8799c8df4b7589869baffb93b68d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "f8c99fddf5974401b64aa8601605b2b2" member_type: VOTER last_known_addr { host: "127.16.94.133" port: 42559 } attrs { promote: false } }
I20240503 14:13:41.181613 20259 raft_consensus.cc:1261] T d4743d5a73c54086b277f4c4b1a248fd P a5d66df07715448fab64928f5bc8299a [term 2 FOLLOWER]: Refusing update from remote peer 64ad8799c8df4b7589869baffb93b68d: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240503 14:13:41.182716 20918 consensus_queue.cc:1035] T d4743d5a73c54086b277f4c4b1a248fd P 64ad8799c8df4b7589869baffb93b68d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 14:13:41.182334 20734 raft_consensus.cc:1261] T d4743d5a73c54086b277f4c4b1a248fd P f8c99fddf5974401b64aa8601605b2b2 [term 2 FOLLOWER]: Refusing update from remote peer 64ad8799c8df4b7589869baffb93b68d: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240503 14:13:41.183637 20918 consensus_queue.cc:1035] T d4743d5a73c54086b277f4c4b1a248fd P 64ad8799c8df4b7589869baffb93b68d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c99fddf5974401b64aa8601605b2b2" member_type: VOTER last_known_addr { host: "127.16.94.133" port: 42559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 14:13:41.185350 20941 ts_tablet_manager.cc:1888] T 904450ab5e8b4250b7af18b98d2b0985 P 64ad8799c8df4b7589869baffb93b68d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.29
I20240503 14:13:41.185458 20941 log.cc:1196] T 904450ab5e8b4250b7af18b98d2b0985 P 64ad8799c8df4b7589869baffb93b68d: Deleting WAL directory at /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714745515241994-16762-0/minicluster-data/ts-0-root/wals/904450ab5e8b4250b7af18b98d2b0985
I20240503 14:13:41.186358 20066 catalog_manager.cc:4830] TS 64ad8799c8df4b7589869baffb93b68d (127.16.94.129:37429): tablet 904450ab5e8b4250b7af18b98d2b0985 (table test-workload [id=cc8b29b2b6ee4a4eb4823287254752f5]) successfully deleted
I20240503 14:13:41.186633 20901 raft_consensus.cc:2932] T d4743d5a73c54086b277f4c4b1a248fd P 64ad8799c8df4b7589869baffb93b68d [term 2 LEADER]: Committing config change with OpId 2.30: config changed from index 29 to 30, VOTER fceaf96557de46c3986c306fa4ac3b9e (127.16.94.131) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "f8c99fddf5974401b64aa8601605b2b2" member_type: VOTER last_known_addr { host: "127.16.94.133" port: 42559 } attrs { promote: false } } }
I20240503 14:13:41.187116 20261 raft_consensus.cc:2932] T d4743d5a73c54086b277f4c4b1a248fd P a5d66df07715448fab64928f5bc8299a [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 29 to 30, VOTER fceaf96557de46c3986c306fa4ac3b9e (127.16.94.131) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "f8c99fddf5974401b64aa8601605b2b2" member_type: VOTER last_known_addr { host: "127.16.94.133" port: 42559 } attrs { promote: false } } }
I20240503 14:13:41.189257 20066 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d4743d5a73c54086b277f4c4b1a248fd with cas_config_opid_index 29: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:41.190940 20081 catalog_manager.cc:5484] T d4743d5a73c54086b277f4c4b1a248fd P 64ad8799c8df4b7589869baffb93b68d reported cstate change: config changed from index 29 to 30, VOTER fceaf96557de46c3986c306fa4ac3b9e (127.16.94.131) evicted. New cstate: current_term: 2 leader_uuid: "64ad8799c8df4b7589869baffb93b68d" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8c99fddf5974401b64aa8601605b2b2" member_type: VOTER last_known_addr { host: "127.16.94.133" port: 42559 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:41.191443 20734 raft_consensus.cc:2932] T d4743d5a73c54086b277f4c4b1a248fd P f8c99fddf5974401b64aa8601605b2b2 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 29 to 30, VOTER fceaf96557de46c3986c306fa4ac3b9e (127.16.94.131) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "f8c99fddf5974401b64aa8601605b2b2" member_type: VOTER last_known_addr { host: "127.16.94.133" port: 42559 } attrs { promote: false } } }
I20240503 14:13:41.195859 20325 tablet_service.cc:1514] Processing DeleteTablet for tablet d4743d5a73c54086b277f4c4b1a248fd with delete_type TABLET_DATA_TOMBSTONED (TS fceaf96557de46c3986c306fa4ac3b9e not found in new config with opid_index 30) from {username='slave'} at 127.0.0.1:56202
I20240503 14:13:41.198693 20942 tablet_replica.cc:330] T d4743d5a73c54086b277f4c4b1a248fd P fceaf96557de46c3986c306fa4ac3b9e: stopping tablet replica
I20240503 14:13:41.198853 20942 raft_consensus.cc:2229] T d4743d5a73c54086b277f4c4b1a248fd P fceaf96557de46c3986c306fa4ac3b9e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:41.199165 20942 raft_consensus.cc:2258] T d4743d5a73c54086b277f4c4b1a248fd P fceaf96557de46c3986c306fa4ac3b9e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:41.200048 20942 ts_tablet_manager.cc:1875] T d4743d5a73c54086b277f4c4b1a248fd P fceaf96557de46c3986c306fa4ac3b9e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:41.205235 20942 ts_tablet_manager.cc:1888] T d4743d5a73c54086b277f4c4b1a248fd P fceaf96557de46c3986c306fa4ac3b9e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.29
I20240503 14:13:41.205319 20942 log.cc:1196] T d4743d5a73c54086b277f4c4b1a248fd P fceaf96557de46c3986c306fa4ac3b9e: Deleting WAL directory at /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714745515241994-16762-0/minicluster-data/ts-2-root/wals/d4743d5a73c54086b277f4c4b1a248fd
I20240503 14:13:41.205879 20068 catalog_manager.cc:4830] TS fceaf96557de46c3986c306fa4ac3b9e (127.16.94.131:39581): tablet d4743d5a73c54086b277f4c4b1a248fd (table test-workload [id=cc8b29b2b6ee4a4eb4823287254752f5]) successfully deleted
I20240503 14:13:41.214241 20943 raft_consensus.cc:1050] T 3054a11a8da4470ea5241b92d164d6a8 P fceaf96557de46c3986c306fa4ac3b9e: attempting to promote NON_VOTER d6bcbc9fecfb44fa99a81917ea2d559a to VOTER
I20240503 14:13:41.214843 20943 consensus_queue.cc:237] T 3054a11a8da4470ea5241b92d164d6a8 P fceaf96557de46c3986c306fa4ac3b9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } attrs { replace: true } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "d6bcbc9fecfb44fa99a81917ea2d559a" member_type: VOTER last_known_addr { host: "127.16.94.134" port: 34983 } attrs { promote: false } }
I20240503 14:13:41.216822 20807 raft_consensus.cc:1261] T 3054a11a8da4470ea5241b92d164d6a8 P d6bcbc9fecfb44fa99a81917ea2d559a [term 2 LEARNER]: Refusing update from remote peer fceaf96557de46c3986c306fa4ac3b9e: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240503 14:13:41.216898 20261 raft_consensus.cc:1261] T 3054a11a8da4470ea5241b92d164d6a8 P a5d66df07715448fab64928f5bc8299a [term 2 FOLLOWER]: Refusing update from remote peer fceaf96557de46c3986c306fa4ac3b9e: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240503 14:13:41.216974 20180 raft_consensus.cc:1261] T 3054a11a8da4470ea5241b92d164d6a8 P 64ad8799c8df4b7589869baffb93b68d [term 2 FOLLOWER]: Refusing update from remote peer fceaf96557de46c3986c306fa4ac3b9e: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240503 14:13:41.217265 20943 consensus_queue.cc:1035] T 3054a11a8da4470ea5241b92d164d6a8 P fceaf96557de46c3986c306fa4ac3b9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6bcbc9fecfb44fa99a81917ea2d559a" member_type: VOTER last_known_addr { host: "127.16.94.134" port: 34983 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 14:13:41.217541 20916 consensus_queue.cc:1035] T 3054a11a8da4470ea5241b92d164d6a8 P fceaf96557de46c3986c306fa4ac3b9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 14:13:41.217872 20944 consensus_queue.cc:1035] T 3054a11a8da4470ea5241b92d164d6a8 P fceaf96557de46c3986c306fa4ac3b9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 14:13:41.224278 20944 raft_consensus.cc:2932] T 3054a11a8da4470ea5241b92d164d6a8 P fceaf96557de46c3986c306fa4ac3b9e [term 2 LEADER]: Committing config change with OpId 2.28: config changed from index 27 to 28, d6bcbc9fecfb44fa99a81917ea2d559a (127.16.94.134) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } attrs { replace: true } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "d6bcbc9fecfb44fa99a81917ea2d559a" member_type: VOTER last_known_addr { host: "127.16.94.134" port: 34983 } attrs { promote: false } } }
I20240503 14:13:41.224612 20180 raft_consensus.cc:2932] T 3054a11a8da4470ea5241b92d164d6a8 P 64ad8799c8df4b7589869baffb93b68d [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index 27 to 28, d6bcbc9fecfb44fa99a81917ea2d559a (127.16.94.134) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } attrs { replace: true } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "d6bcbc9fecfb44fa99a81917ea2d559a" member_type: VOTER last_known_addr { host: "127.16.94.134" port: 34983 } attrs { promote: false } } }
I20240503 14:13:41.224612 20261 raft_consensus.cc:2932] T 3054a11a8da4470ea5241b92d164d6a8 P a5d66df07715448fab64928f5bc8299a [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index 27 to 28, d6bcbc9fecfb44fa99a81917ea2d559a (127.16.94.134) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } attrs { replace: true } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "d6bcbc9fecfb44fa99a81917ea2d559a" member_type: VOTER last_known_addr { host: "127.16.94.134" port: 34983 } attrs { promote: false } } }
I20240503 14:13:41.229486 20078 catalog_manager.cc:5484] T 3054a11a8da4470ea5241b92d164d6a8 P 64ad8799c8df4b7589869baffb93b68d reported cstate change: config changed from index 27 to 28, d6bcbc9fecfb44fa99a81917ea2d559a (127.16.94.134) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "fceaf96557de46c3986c306fa4ac3b9e" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } attrs { replace: true } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "d6bcbc9fecfb44fa99a81917ea2d559a" member_type: VOTER last_known_addr { host: "127.16.94.134" port: 34983 } attrs { promote: false } } }
I20240503 14:13:41.229960 20807 raft_consensus.cc:2932] T 3054a11a8da4470ea5241b92d164d6a8 P d6bcbc9fecfb44fa99a81917ea2d559a [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index 27 to 28, d6bcbc9fecfb44fa99a81917ea2d559a (127.16.94.134) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } attrs { replace: true } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "d6bcbc9fecfb44fa99a81917ea2d559a" member_type: VOTER last_known_addr { host: "127.16.94.134" port: 34983 } attrs { promote: false } } }
I20240503 14:13:41.237323 20330 consensus_queue.cc:237] T 3054a11a8da4470ea5241b92d164d6a8 P fceaf96557de46c3986c306fa4ac3b9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "d6bcbc9fecfb44fa99a81917ea2d559a" member_type: VOTER last_known_addr { host: "127.16.94.134" port: 34983 } attrs { promote: false } }
I20240503 14:13:41.239135 20807 raft_consensus.cc:1261] T 3054a11a8da4470ea5241b92d164d6a8 P d6bcbc9fecfb44fa99a81917ea2d559a [term 2 FOLLOWER]: Refusing update from remote peer fceaf96557de46c3986c306fa4ac3b9e: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240503 14:13:41.239135 20180 raft_consensus.cc:1261] T 3054a11a8da4470ea5241b92d164d6a8 P 64ad8799c8df4b7589869baffb93b68d [term 2 FOLLOWER]: Refusing update from remote peer fceaf96557de46c3986c306fa4ac3b9e: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240503 14:13:41.239778 20949 consensus_queue.cc:1035] T 3054a11a8da4470ea5241b92d164d6a8 P fceaf96557de46c3986c306fa4ac3b9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6bcbc9fecfb44fa99a81917ea2d559a" member_type: VOTER last_known_addr { host: "127.16.94.134" port: 34983 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 14:13:41.240065 20944 consensus_queue.cc:1035] T 3054a11a8da4470ea5241b92d164d6a8 P fceaf96557de46c3986c306fa4ac3b9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 14:13:41.242035 20943 raft_consensus.cc:2932] T 3054a11a8da4470ea5241b92d164d6a8 P fceaf96557de46c3986c306fa4ac3b9e [term 2 LEADER]: Committing config change with OpId 2.29: config changed from index 28 to 29, VOTER a5d66df07715448fab64928f5bc8299a (127.16.94.130) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "d6bcbc9fecfb44fa99a81917ea2d559a" member_type: VOTER last_known_addr { host: "127.16.94.134" port: 34983 } attrs { promote: false } } }
I20240503 14:13:41.242712 20180 raft_consensus.cc:2932] T 3054a11a8da4470ea5241b92d164d6a8 P 64ad8799c8df4b7589869baffb93b68d [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 28 to 29, VOTER a5d66df07715448fab64928f5bc8299a (127.16.94.130) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "d6bcbc9fecfb44fa99a81917ea2d559a" member_type: VOTER last_known_addr { host: "127.16.94.134" port: 34983 } attrs { promote: false } } }
I20240503 14:13:41.242589 20807 raft_consensus.cc:2932] T 3054a11a8da4470ea5241b92d164d6a8 P d6bcbc9fecfb44fa99a81917ea2d559a [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 28 to 29, VOTER a5d66df07715448fab64928f5bc8299a (127.16.94.130) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "d6bcbc9fecfb44fa99a81917ea2d559a" member_type: VOTER last_known_addr { host: "127.16.94.134" port: 34983 } attrs { promote: false } } }
I20240503 14:13:41.246625 20068 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3054a11a8da4470ea5241b92d164d6a8 with cas_config_opid_index 28: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:41.246562 20078 catalog_manager.cc:5484] T 3054a11a8da4470ea5241b92d164d6a8 P fceaf96557de46c3986c306fa4ac3b9e reported cstate change: config changed from index 28 to 29, VOTER a5d66df07715448fab64928f5bc8299a (127.16.94.130) evicted. New cstate: current_term: 2 leader_uuid: "fceaf96557de46c3986c306fa4ac3b9e" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6bcbc9fecfb44fa99a81917ea2d559a" member_type: VOTER last_known_addr { host: "127.16.94.134" port: 34983 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:41.250207 20249 tablet_service.cc:1514] Processing DeleteTablet for tablet 3054a11a8da4470ea5241b92d164d6a8 with delete_type TABLET_DATA_TOMBSTONED (TS a5d66df07715448fab64928f5bc8299a not found in new config with opid_index 29) from {username='slave'} at 127.0.0.1:38642
I20240503 14:13:41.250487 20929 tablet_replica.cc:330] T 3054a11a8da4470ea5241b92d164d6a8 P a5d66df07715448fab64928f5bc8299a: stopping tablet replica
I20240503 14:13:41.250667 20929 raft_consensus.cc:2229] T 3054a11a8da4470ea5241b92d164d6a8 P a5d66df07715448fab64928f5bc8299a [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:41.250912 20929 raft_consensus.cc:2258] T 3054a11a8da4470ea5241b92d164d6a8 P a5d66df07715448fab64928f5bc8299a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:41.251974 20929 ts_tablet_manager.cc:1875] T 3054a11a8da4470ea5241b92d164d6a8 P a5d66df07715448fab64928f5bc8299a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:41.257820 20929 ts_tablet_manager.cc:1888] T 3054a11a8da4470ea5241b92d164d6a8 P a5d66df07715448fab64928f5bc8299a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.28
I20240503 14:13:41.257938 20929 log.cc:1196] T 3054a11a8da4470ea5241b92d164d6a8 P a5d66df07715448fab64928f5bc8299a: Deleting WAL directory at /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714745515241994-16762-0/minicluster-data/ts-1-root/wals/3054a11a8da4470ea5241b92d164d6a8
I20240503 14:13:41.258576 20069 catalog_manager.cc:4830] TS a5d66df07715448fab64928f5bc8299a (127.16.94.130:39295): tablet 3054a11a8da4470ea5241b92d164d6a8 (table test-workload [id=cc8b29b2b6ee4a4eb4823287254752f5]) successfully deleted
I20240503 14:13:41.342885 20949 raft_consensus.cc:1050] T 0eb7552695304a1ea6f8e02f189f7295 P fceaf96557de46c3986c306fa4ac3b9e: attempting to promote NON_VOTER d6bcbc9fecfb44fa99a81917ea2d559a to VOTER
I20240503 14:13:41.343611 20949 consensus_queue.cc:237] T 0eb7552695304a1ea6f8e02f189f7295 P fceaf96557de46c3986c306fa4ac3b9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } attrs { replace: true } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "d6bcbc9fecfb44fa99a81917ea2d559a" member_type: VOTER last_known_addr { host: "127.16.94.134" port: 34983 } attrs { promote: false } }
I20240503 14:13:41.345736 20180 raft_consensus.cc:1261] T 0eb7552695304a1ea6f8e02f189f7295 P 64ad8799c8df4b7589869baffb93b68d [term 1 FOLLOWER]: Refusing update from remote peer fceaf96557de46c3986c306fa4ac3b9e: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 14:13:41.345770 20807 raft_consensus.cc:1261] T 0eb7552695304a1ea6f8e02f189f7295 P d6bcbc9fecfb44fa99a81917ea2d559a [term 1 LEARNER]: Refusing update from remote peer fceaf96557de46c3986c306fa4ac3b9e: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 14:13:41.345770 20261 raft_consensus.cc:1261] T 0eb7552695304a1ea6f8e02f189f7295 P a5d66df07715448fab64928f5bc8299a [term 1 FOLLOWER]: Refusing update from remote peer fceaf96557de46c3986c306fa4ac3b9e: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 14:13:41.346252 20949 consensus_queue.cc:1035] T 0eb7552695304a1ea6f8e02f189f7295 P fceaf96557de46c3986c306fa4ac3b9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 14:13:41.346637 20944 consensus_queue.cc:1035] T 0eb7552695304a1ea6f8e02f189f7295 P fceaf96557de46c3986c306fa4ac3b9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6bcbc9fecfb44fa99a81917ea2d559a" member_type: VOTER last_known_addr { host: "127.16.94.134" port: 34983 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 14:13:41.346931 20916 consensus_queue.cc:1035] T 0eb7552695304a1ea6f8e02f189f7295 P fceaf96557de46c3986c306fa4ac3b9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 14:13:41.351058 20949 raft_consensus.cc:2932] T 0eb7552695304a1ea6f8e02f189f7295 P fceaf96557de46c3986c306fa4ac3b9e [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index 27 to 28, d6bcbc9fecfb44fa99a81917ea2d559a (127.16.94.134) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } attrs { replace: true } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "d6bcbc9fecfb44fa99a81917ea2d559a" member_type: VOTER last_known_addr { host: "127.16.94.134" port: 34983 } attrs { promote: false } } }
I20240503 14:13:41.351843 20261 raft_consensus.cc:2932] T 0eb7552695304a1ea6f8e02f189f7295 P a5d66df07715448fab64928f5bc8299a [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, d6bcbc9fecfb44fa99a81917ea2d559a (127.16.94.134) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } attrs { replace: true } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "d6bcbc9fecfb44fa99a81917ea2d559a" member_type: VOTER last_known_addr { host: "127.16.94.134" port: 34983 } attrs { promote: false } } }
I20240503 14:13:41.354887 20179 raft_consensus.cc:2932] T 0eb7552695304a1ea6f8e02f189f7295 P 64ad8799c8df4b7589869baffb93b68d [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, d6bcbc9fecfb44fa99a81917ea2d559a (127.16.94.134) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } attrs { replace: true } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "d6bcbc9fecfb44fa99a81917ea2d559a" member_type: VOTER last_known_addr { host: "127.16.94.134" port: 34983 } attrs { promote: false } } }
I20240503 14:13:41.355692 20807 raft_consensus.cc:2932] T 0eb7552695304a1ea6f8e02f189f7295 P d6bcbc9fecfb44fa99a81917ea2d559a [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, d6bcbc9fecfb44fa99a81917ea2d559a (127.16.94.134) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } attrs { replace: true } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "d6bcbc9fecfb44fa99a81917ea2d559a" member_type: VOTER last_known_addr { host: "127.16.94.134" port: 34983 } attrs { promote: false } } }
I20240503 14:13:41.355808 20077 catalog_manager.cc:5484] T 0eb7552695304a1ea6f8e02f189f7295 P fceaf96557de46c3986c306fa4ac3b9e reported cstate change: config changed from index 27 to 28, d6bcbc9fecfb44fa99a81917ea2d559a (127.16.94.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fceaf96557de46c3986c306fa4ac3b9e" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6bcbc9fecfb44fa99a81917ea2d559a" member_type: VOTER last_known_addr { host: "127.16.94.134" port: 34983 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:41.361681 20330 consensus_queue.cc:237] T 0eb7552695304a1ea6f8e02f189f7295 P fceaf96557de46c3986c306fa4ac3b9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "d6bcbc9fecfb44fa99a81917ea2d559a" member_type: VOTER last_known_addr { host: "127.16.94.134" port: 34983 } attrs { promote: false } }
I20240503 14:13:41.363412 20261 raft_consensus.cc:1261] T 0eb7552695304a1ea6f8e02f189f7295 P a5d66df07715448fab64928f5bc8299a [term 1 FOLLOWER]: Refusing update from remote peer fceaf96557de46c3986c306fa4ac3b9e: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240503 14:13:41.364096 20949 consensus_queue.cc:1035] T 0eb7552695304a1ea6f8e02f189f7295 P fceaf96557de46c3986c306fa4ac3b9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 14:13:41.364431 20807 raft_consensus.cc:1261] T 0eb7552695304a1ea6f8e02f189f7295 P d6bcbc9fecfb44fa99a81917ea2d559a [term 1 FOLLOWER]: Refusing update from remote peer fceaf96557de46c3986c306fa4ac3b9e: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240503 14:13:41.364975 20949 consensus_queue.cc:1035] T 0eb7552695304a1ea6f8e02f189f7295 P fceaf96557de46c3986c306fa4ac3b9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6bcbc9fecfb44fa99a81917ea2d559a" member_type: VOTER last_known_addr { host: "127.16.94.134" port: 34983 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 14:13:41.366672 20944 raft_consensus.cc:2932] T 0eb7552695304a1ea6f8e02f189f7295 P fceaf96557de46c3986c306fa4ac3b9e [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER 64ad8799c8df4b7589869baffb93b68d (127.16.94.129) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "d6bcbc9fecfb44fa99a81917ea2d559a" member_type: VOTER last_known_addr { host: "127.16.94.134" port: 34983 } attrs { promote: false } } }
I20240503 14:13:41.368539 20807 raft_consensus.cc:2932] T 0eb7552695304a1ea6f8e02f189f7295 P d6bcbc9fecfb44fa99a81917ea2d559a [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER 64ad8799c8df4b7589869baffb93b68d (127.16.94.129) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "d6bcbc9fecfb44fa99a81917ea2d559a" member_type: VOTER last_known_addr { host: "127.16.94.134" port: 34983 } attrs { promote: false } } }
I20240503 14:13:41.371304 20068 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0eb7552695304a1ea6f8e02f189f7295 with cas_config_opid_index 28: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:41.370572 20261 raft_consensus.cc:2932] T 0eb7552695304a1ea6f8e02f189f7295 P a5d66df07715448fab64928f5bc8299a [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER 64ad8799c8df4b7589869baffb93b68d (127.16.94.129) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "d6bcbc9fecfb44fa99a81917ea2d559a" member_type: VOTER last_known_addr { host: "127.16.94.134" port: 34983 } attrs { promote: false } } }
I20240503 14:13:41.371726 20078 catalog_manager.cc:5484] T 0eb7552695304a1ea6f8e02f189f7295 P fceaf96557de46c3986c306fa4ac3b9e reported cstate change: config changed from index 28 to 29, VOTER 64ad8799c8df4b7589869baffb93b68d (127.16.94.129) evicted. New cstate: current_term: 1 leader_uuid: "fceaf96557de46c3986c306fa4ac3b9e" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6bcbc9fecfb44fa99a81917ea2d559a" member_type: VOTER last_known_addr { host: "127.16.94.134" port: 34983 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:41.375232 20172 tablet_service.cc:1514] Processing DeleteTablet for tablet 0eb7552695304a1ea6f8e02f189f7295 with delete_type TABLET_DATA_TOMBSTONED (TS 64ad8799c8df4b7589869baffb93b68d not found in new config with opid_index 29) from {username='slave'} at 127.0.0.1:54496
I20240503 14:13:41.375502 20941 tablet_replica.cc:330] T 0eb7552695304a1ea6f8e02f189f7295 P 64ad8799c8df4b7589869baffb93b68d: stopping tablet replica
I20240503 14:13:41.375671 20941 raft_consensus.cc:2229] T 0eb7552695304a1ea6f8e02f189f7295 P 64ad8799c8df4b7589869baffb93b68d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:41.375985 20941 raft_consensus.cc:2258] T 0eb7552695304a1ea6f8e02f189f7295 P 64ad8799c8df4b7589869baffb93b68d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:41.376658 20941 ts_tablet_manager.cc:1875] T 0eb7552695304a1ea6f8e02f189f7295 P 64ad8799c8df4b7589869baffb93b68d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:41.381841 20941 ts_tablet_manager.cc:1888] T 0eb7552695304a1ea6f8e02f189f7295 P 64ad8799c8df4b7589869baffb93b68d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.28
I20240503 14:13:41.381939 20941 log.cc:1196] T 0eb7552695304a1ea6f8e02f189f7295 P 64ad8799c8df4b7589869baffb93b68d: Deleting WAL directory at /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714745515241994-16762-0/minicluster-data/ts-0-root/wals/0eb7552695304a1ea6f8e02f189f7295
I20240503 14:13:41.382619 20066 catalog_manager.cc:4830] TS 64ad8799c8df4b7589869baffb93b68d (127.16.94.129:37429): tablet 0eb7552695304a1ea6f8e02f189f7295 (table test-workload [id=cc8b29b2b6ee4a4eb4823287254752f5]) successfully deleted
I20240503 14:13:41.401816 20921 raft_consensus.cc:1050] T 220ef8bab4694ace9bdf0fce06226efa P 64ad8799c8df4b7589869baffb93b68d: attempting to promote NON_VOTER f8c99fddf5974401b64aa8601605b2b2 to VOTER
I20240503 14:13:41.402724 20921 consensus_queue.cc:237] T 220ef8bab4694ace9bdf0fce06226efa P 64ad8799c8df4b7589869baffb93b68d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } attrs { replace: true } } peers { permanent_uuid: "f8c99fddf5974401b64aa8601605b2b2" member_type: VOTER last_known_addr { host: "127.16.94.133" port: 42559 } attrs { promote: false } }
I20240503 14:13:41.405020 20329 raft_consensus.cc:1261] T 220ef8bab4694ace9bdf0fce06226efa P fceaf96557de46c3986c306fa4ac3b9e [term 1 FOLLOWER]: Refusing update from remote peer 64ad8799c8df4b7589869baffb93b68d: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240503 14:13:41.405154 20261 raft_consensus.cc:1261] T 220ef8bab4694ace9bdf0fce06226efa P a5d66df07715448fab64928f5bc8299a [term 1 FOLLOWER]: Refusing update from remote peer 64ad8799c8df4b7589869baffb93b68d: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240503 14:13:41.405802 20734 raft_consensus.cc:1261] T 220ef8bab4694ace9bdf0fce06226efa P f8c99fddf5974401b64aa8601605b2b2 [term 1 LEARNER]: Refusing update from remote peer 64ad8799c8df4b7589869baffb93b68d: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240503 14:13:41.405771 20918 consensus_queue.cc:1035] T 220ef8bab4694ace9bdf0fce06226efa P 64ad8799c8df4b7589869baffb93b68d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 14:13:41.406275 20921 consensus_queue.cc:1035] T 220ef8bab4694ace9bdf0fce06226efa P 64ad8799c8df4b7589869baffb93b68d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c99fddf5974401b64aa8601605b2b2" member_type: VOTER last_known_addr { host: "127.16.94.133" port: 42559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 14:13:41.407065 20919 consensus_queue.cc:1035] T 220ef8bab4694ace9bdf0fce06226efa P 64ad8799c8df4b7589869baffb93b68d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 14:13:41.411244 20919 raft_consensus.cc:2932] T 220ef8bab4694ace9bdf0fce06226efa P 64ad8799c8df4b7589869baffb93b68d [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, f8c99fddf5974401b64aa8601605b2b2 (127.16.94.133) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } attrs { replace: true } } peers { permanent_uuid: "f8c99fddf5974401b64aa8601605b2b2" member_type: VOTER last_known_addr { host: "127.16.94.133" port: 42559 } attrs { promote: false } } }
I20240503 14:13:41.414815 20734 raft_consensus.cc:2932] T 220ef8bab4694ace9bdf0fce06226efa P f8c99fddf5974401b64aa8601605b2b2 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, f8c99fddf5974401b64aa8601605b2b2 (127.16.94.133) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } attrs { replace: true } } peers { permanent_uuid: "f8c99fddf5974401b64aa8601605b2b2" member_type: VOTER last_known_addr { host: "127.16.94.133" port: 42559 } attrs { promote: false } } }
I20240503 14:13:41.416498 20261 raft_consensus.cc:2932] T 220ef8bab4694ace9bdf0fce06226efa P a5d66df07715448fab64928f5bc8299a [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, f8c99fddf5974401b64aa8601605b2b2 (127.16.94.133) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } attrs { replace: true } } peers { permanent_uuid: "f8c99fddf5974401b64aa8601605b2b2" member_type: VOTER last_known_addr { host: "127.16.94.133" port: 42559 } attrs { promote: false } } }
I20240503 14:13:41.422387 20078 catalog_manager.cc:5484] T 220ef8bab4694ace9bdf0fce06226efa P 64ad8799c8df4b7589869baffb93b68d reported cstate change: config changed from index 28 to 29, f8c99fddf5974401b64aa8601605b2b2 (127.16.94.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "64ad8799c8df4b7589869baffb93b68d" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8c99fddf5974401b64aa8601605b2b2" member_type: VOTER last_known_addr { host: "127.16.94.133" port: 42559 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:41.428382 20179 consensus_queue.cc:237] T 220ef8bab4694ace9bdf0fce06226efa P 64ad8799c8df4b7589869baffb93b68d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "f8c99fddf5974401b64aa8601605b2b2" member_type: VOTER last_known_addr { host: "127.16.94.133" port: 42559 } attrs { promote: false } }
I20240503 14:13:41.430203 20734 raft_consensus.cc:1261] T 220ef8bab4694ace9bdf0fce06226efa P f8c99fddf5974401b64aa8601605b2b2 [term 1 FOLLOWER]: Refusing update from remote peer 64ad8799c8df4b7589869baffb93b68d: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240503 14:13:41.430420 20261 raft_consensus.cc:1261] T 220ef8bab4694ace9bdf0fce06226efa P a5d66df07715448fab64928f5bc8299a [term 1 FOLLOWER]: Refusing update from remote peer 64ad8799c8df4b7589869baffb93b68d: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240503 14:13:41.430846 20919 consensus_queue.cc:1035] T 220ef8bab4694ace9bdf0fce06226efa P 64ad8799c8df4b7589869baffb93b68d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 14:13:41.431021 20921 consensus_queue.cc:1035] T 220ef8bab4694ace9bdf0fce06226efa P 64ad8799c8df4b7589869baffb93b68d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c99fddf5974401b64aa8601605b2b2" member_type: VOTER last_known_addr { host: "127.16.94.133" port: 42559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 14:13:41.433755 20919 raft_consensus.cc:2932] T 220ef8bab4694ace9bdf0fce06226efa P 64ad8799c8df4b7589869baffb93b68d [term 1 LEADER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER fceaf96557de46c3986c306fa4ac3b9e (127.16.94.131) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "f8c99fddf5974401b64aa8601605b2b2" member_type: VOTER last_known_addr { host: "127.16.94.133" port: 42559 } attrs { promote: false } } }
I20240503 14:13:41.435312 20734 raft_consensus.cc:2932] T 220ef8bab4694ace9bdf0fce06226efa P f8c99fddf5974401b64aa8601605b2b2 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER fceaf96557de46c3986c306fa4ac3b9e (127.16.94.131) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "f8c99fddf5974401b64aa8601605b2b2" member_type: VOTER last_known_addr { host: "127.16.94.133" port: 42559 } attrs { promote: false } } }
I20240503 14:13:41.434334 20261 raft_consensus.cc:2932] T 220ef8bab4694ace9bdf0fce06226efa P a5d66df07715448fab64928f5bc8299a [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER fceaf96557de46c3986c306fa4ac3b9e (127.16.94.131) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "f8c99fddf5974401b64aa8601605b2b2" member_type: VOTER last_known_addr { host: "127.16.94.133" port: 42559 } attrs { promote: false } } }
I20240503 14:13:41.438941 20066 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 220ef8bab4694ace9bdf0fce06226efa with cas_config_opid_index 29: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:41.440183 20081 catalog_manager.cc:5484] T 220ef8bab4694ace9bdf0fce06226efa P 64ad8799c8df4b7589869baffb93b68d reported cstate change: config changed from index 29 to 30, VOTER fceaf96557de46c3986c306fa4ac3b9e (127.16.94.131) evicted. New cstate: current_term: 1 leader_uuid: "64ad8799c8df4b7589869baffb93b68d" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8c99fddf5974401b64aa8601605b2b2" member_type: VOTER last_known_addr { host: "127.16.94.133" port: 42559 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:41.445079 20325 tablet_service.cc:1514] Processing DeleteTablet for tablet 220ef8bab4694ace9bdf0fce06226efa with delete_type TABLET_DATA_TOMBSTONED (TS fceaf96557de46c3986c306fa4ac3b9e not found in new config with opid_index 30) from {username='slave'} at 127.0.0.1:56202
I20240503 14:13:41.445377 20942 tablet_replica.cc:330] T 220ef8bab4694ace9bdf0fce06226efa P fceaf96557de46c3986c306fa4ac3b9e: stopping tablet replica
I20240503 14:13:41.445581 20942 raft_consensus.cc:2229] T 220ef8bab4694ace9bdf0fce06226efa P fceaf96557de46c3986c306fa4ac3b9e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:41.445901 20942 pending_rounds.cc:62] T 220ef8bab4694ace9bdf0fce06226efa P fceaf96557de46c3986c306fa4ac3b9e: Trying to abort 1 pending ops.
I20240503 14:13:41.446043 20942 pending_rounds.cc:68] T 220ef8bab4694ace9bdf0fce06226efa P fceaf96557de46c3986c306fa4ac3b9e: Aborting op as it isn't in flight: id { term: 1 index: 29 } timestamp: 7023598065263534080 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "220ef8bab4694ace9bdf0fce06226efa" old_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } attrs { replace: true } } peers { permanent_uuid: "f8c99fddf5974401b64aa8601605b2b2" member_type: NON_VOTER last_known_addr { host: "127.16.94.133" port: 42559 } attrs { promote: true } } } new_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } attrs { replace: true } } peers { permanent_uuid: "f8c99fddf5974401b64aa8601605b2b2" member_type: VOTER last_known_addr { host: "127.16.94.133" port: 42559 } attrs { promote: false } } } }
I20240503 14:13:41.446465 20942 raft_consensus.cc:2894] T 220ef8bab4694ace9bdf0fce06226efa P fceaf96557de46c3986c306fa4ac3b9e [term 1 FOLLOWER]: Aborting config change with OpId 1.29: Aborted: Op aborted
I20240503 14:13:41.446578 20942 raft_consensus.cc:2866] T 220ef8bab4694ace9bdf0fce06226efa P fceaf96557de46c3986c306fa4ac3b9e [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 14:13:41.446662 20942 raft_consensus.cc:2258] T 220ef8bab4694ace9bdf0fce06226efa P fceaf96557de46c3986c306fa4ac3b9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:41.447422 20942 ts_tablet_manager.cc:1875] T 220ef8bab4694ace9bdf0fce06226efa P fceaf96557de46c3986c306fa4ac3b9e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:41.452504 20942 ts_tablet_manager.cc:1888] T 220ef8bab4694ace9bdf0fce06226efa P fceaf96557de46c3986c306fa4ac3b9e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.29
I20240503 14:13:41.452598 20942 log.cc:1196] T 220ef8bab4694ace9bdf0fce06226efa P fceaf96557de46c3986c306fa4ac3b9e: Deleting WAL directory at /tmp/dist-test-taskc2SgQJ/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714745515241994-16762-0/minicluster-data/ts-2-root/wals/220ef8bab4694ace9bdf0fce06226efa
I20240503 14:13:41.453193 20068 catalog_manager.cc:4830] TS fceaf96557de46c3986c306fa4ac3b9e (127.16.94.131:39581): tablet 220ef8bab4694ace9bdf0fce06226efa (table test-workload [id=cc8b29b2b6ee4a4eb4823287254752f5]) successfully deleted
I20240503 14:13:42.292519 20136 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:13:42.305393 20330 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0eb7552695304a1ea6f8e02f189f7295"
dest_uuid: "fceaf96557de46c3986c306fa4ac3b9e"
mode: GRACEFUL
new_leader_uuid: "d6bcbc9fecfb44fa99a81917ea2d559a"
from {username='slave'} at 127.0.0.1:56260
I20240503 14:13:42.305605 20330 raft_consensus.cc:595] T 0eb7552695304a1ea6f8e02f189f7295 P fceaf96557de46c3986c306fa4ac3b9e [term 1 LEADER]: Received request to transfer leadership to TS d6bcbc9fecfb44fa99a81917ea2d559a
I20240503 14:13:42.310793 20180 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "220ef8bab4694ace9bdf0fce06226efa"
dest_uuid: "64ad8799c8df4b7589869baffb93b68d"
mode: GRACEFUL
new_leader_uuid: "f8c99fddf5974401b64aa8601605b2b2"
from {username='slave'} at 127.0.0.1:54578
I20240503 14:13:42.311014 20180 raft_consensus.cc:595] T 220ef8bab4694ace9bdf0fce06226efa P 64ad8799c8df4b7589869baffb93b68d [term 1 LEADER]: Received request to transfer leadership to TS f8c99fddf5974401b64aa8601605b2b2
I20240503 14:13:42.311880 20180 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "29f4ee03917646f6a5374217bed7a1be"
dest_uuid: "64ad8799c8df4b7589869baffb93b68d"
mode: GRACEFUL
new_leader_uuid: "c4d0f91a34804de99e3a2a255cd06aed"
from {username='slave'} at 127.0.0.1:54578
I20240503 14:13:42.312022 20180 raft_consensus.cc:595] T 29f4ee03917646f6a5374217bed7a1be P 64ad8799c8df4b7589869baffb93b68d [term 1 LEADER]: Received request to transfer leadership to TS c4d0f91a34804de99e3a2a255cd06aed
I20240503 14:13:42.312794 20261 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7905fd813a2d4a8aa8b2ac1e752b37dc"
dest_uuid: "a5d66df07715448fab64928f5bc8299a"
mode: GRACEFUL
new_leader_uuid: "64ad8799c8df4b7589869baffb93b68d"
from {username='slave'} at 127.0.0.1:38684
I20240503 14:13:42.312916 20261 raft_consensus.cc:595] T 7905fd813a2d4a8aa8b2ac1e752b37dc P a5d66df07715448fab64928f5bc8299a [term 1 LEADER]: Received request to transfer leadership to TS 64ad8799c8df4b7589869baffb93b68d
I20240503 14:13:42.313692 20330 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ad711d78ee524d979867b65906abe456"
dest_uuid: "fceaf96557de46c3986c306fa4ac3b9e"
mode: GRACEFUL
new_leader_uuid: "a5d66df07715448fab64928f5bc8299a"
from {username='slave'} at 127.0.0.1:56260
I20240503 14:13:42.313807 20330 raft_consensus.cc:595] T ad711d78ee524d979867b65906abe456 P fceaf96557de46c3986c306fa4ac3b9e [term 1 LEADER]: Received request to transfer leadership to TS a5d66df07715448fab64928f5bc8299a
I20240503 14:13:42.314522 20261 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c45ceab3a36d4225aed74654ed530c9d"
dest_uuid: "a5d66df07715448fab64928f5bc8299a"
mode: GRACEFUL
new_leader_uuid: "64ad8799c8df4b7589869baffb93b68d"
from {username='slave'} at 127.0.0.1:38684
I20240503 14:13:42.314656 20261 raft_consensus.cc:595] T c45ceab3a36d4225aed74654ed530c9d P a5d66df07715448fab64928f5bc8299a [term 1 LEADER]: Received request to transfer leadership to TS 64ad8799c8df4b7589869baffb93b68d
I20240503 14:13:42.314939 20136 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 6
I20240503 14:13:42.315114 20136 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
W20240503 14:13:42.449956 20131 auto_rebalancer.cc:254] failed to send replica move request: Remote error: Service unavailable: leader transfer in progress
W20240503 14:13:42.451377 20131 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 29f4ee03917646f6a5374217bed7a1be, TS fceaf96557de46c3986c306fa4ac3b9e -> TS f8c99fddf5974401b64aa8601605b2b2 move failed, destination replica disappeared from tablet's Raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "c4d0f91a34804de99e3a2a255cd06aed" member_type: VOTER last_known_addr { host: "127.16.94.132" port: 41945 } attrs { promote: false } }
W20240503 14:13:42.451460 20131 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 29f4ee03917646f6a5374217bed7a1be, TS fceaf96557de46c3986c306fa4ac3b9e -> TS f8c99fddf5974401b64aa8601605b2b2 move failed, destination replica disappeared from tablet's Raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "c4d0f91a34804de99e3a2a255cd06aed" member_type: VOTER last_known_addr { host: "127.16.94.132" port: 41945 } attrs { promote: false } }
W20240503 14:13:42.452531 20131 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 7905fd813a2d4a8aa8b2ac1e752b37dc, TS 64ad8799c8df4b7589869baffb93b68d -> TS c4d0f91a34804de99e3a2a255cd06aed move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
W20240503 14:13:42.452607 20131 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 7905fd813a2d4a8aa8b2ac1e752b37dc, TS 64ad8799c8df4b7589869baffb93b68d -> TS c4d0f91a34804de99e3a2a255cd06aed move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
W20240503 14:13:42.453583 20131 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 7a82ae1f4c66455caea352a304af2b51, TS a5d66df07715448fab64928f5bc8299a -> TS d6bcbc9fecfb44fa99a81917ea2d559a move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
W20240503 14:13:42.453639 20131 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 7a82ae1f4c66455caea352a304af2b51, TS a5d66df07715448fab64928f5bc8299a -> TS d6bcbc9fecfb44fa99a81917ea2d559a move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
W20240503 14:13:42.454730 20131 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 44a4d2b820fc4d0da06c659fe98fe57c, TS 64ad8799c8df4b7589869baffb93b68d -> TS d6bcbc9fecfb44fa99a81917ea2d559a move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
W20240503 14:13:42.454787 20131 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 44a4d2b820fc4d0da06c659fe98fe57c, TS 64ad8799c8df4b7589869baffb93b68d -> TS d6bcbc9fecfb44fa99a81917ea2d559a move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
W20240503 14:13:42.455929 20131 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 0eb7552695304a1ea6f8e02f189f7295, TS a5d66df07715448fab64928f5bc8299a -> TS c4d0f91a34804de99e3a2a255cd06aed move failed, destination replica disappeared from tablet's Raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "d6bcbc9fecfb44fa99a81917ea2d559a" member_type: VOTER last_known_addr { host: "127.16.94.134" port: 34983 } attrs { promote: false } }
W20240503 14:13:42.455988 20131 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 0eb7552695304a1ea6f8e02f189f7295, TS a5d66df07715448fab64928f5bc8299a -> TS c4d0f91a34804de99e3a2a255cd06aed move failed, destination replica disappeared from tablet's Raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "d6bcbc9fecfb44fa99a81917ea2d559a" member_type: VOTER last_known_addr { host: "127.16.94.134" port: 34983 } attrs { promote: false } }
W20240503 14:13:42.457027 20131 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet c45ceab3a36d4225aed74654ed530c9d, TS fceaf96557de46c3986c306fa4ac3b9e -> TS f8c99fddf5974401b64aa8601605b2b2 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
W20240503 14:13:42.457082 20131 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet c45ceab3a36d4225aed74654ed530c9d, TS fceaf96557de46c3986c306fa4ac3b9e -> TS f8c99fddf5974401b64aa8601605b2b2 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:42.519505 20958 raft_consensus.cc:979] T 7905fd813a2d4a8aa8b2ac1e752b37dc P a5d66df07715448fab64928f5bc8299a: : Instructing follower 64ad8799c8df4b7589869baffb93b68d to start an election
I20240503 14:13:42.519644 20958 raft_consensus.cc:1067] T 7905fd813a2d4a8aa8b2ac1e752b37dc P a5d66df07715448fab64928f5bc8299a [term 1 LEADER]: Signalling peer 64ad8799c8df4b7589869baffb93b68d to start an election
I20240503 14:13:42.520186 20180 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7905fd813a2d4a8aa8b2ac1e752b37dc"
dest_uuid: "64ad8799c8df4b7589869baffb93b68d"
from {username='slave'} at 127.0.0.1:54502
I20240503 14:13:42.520375 20180 raft_consensus.cc:482] T 7905fd813a2d4a8aa8b2ac1e752b37dc P 64ad8799c8df4b7589869baffb93b68d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:13:42.520467 20180 raft_consensus.cc:3037] T 7905fd813a2d4a8aa8b2ac1e752b37dc P 64ad8799c8df4b7589869baffb93b68d [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:42.521358 20919 raft_consensus.cc:979] T 29f4ee03917646f6a5374217bed7a1be P 64ad8799c8df4b7589869baffb93b68d: : Instructing follower c4d0f91a34804de99e3a2a255cd06aed to start an election
I20240503 14:13:42.521479 20919 raft_consensus.cc:1067] T 29f4ee03917646f6a5374217bed7a1be P 64ad8799c8df4b7589869baffb93b68d [term 1 LEADER]: Signalling peer c4d0f91a34804de99e3a2a255cd06aed to start an election
I20240503 14:13:42.521994 20658 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "29f4ee03917646f6a5374217bed7a1be"
dest_uuid: "c4d0f91a34804de99e3a2a255cd06aed"
from {username='slave'} at 127.0.0.1:49898
I20240503 14:13:42.522166 20658 raft_consensus.cc:482] T 29f4ee03917646f6a5374217bed7a1be P c4d0f91a34804de99e3a2a255cd06aed [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:13:42.522254 20658 raft_consensus.cc:3037] T 29f4ee03917646f6a5374217bed7a1be P c4d0f91a34804de99e3a2a255cd06aed [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:42.523257 20180 raft_consensus.cc:504] T 7905fd813a2d4a8aa8b2ac1e752b37dc P 64ad8799c8df4b7589869baffb93b68d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:42.523864 20180 leader_election.cc:288] T 7905fd813a2d4a8aa8b2ac1e752b37dc P 64ad8799c8df4b7589869baffb93b68d [CANDIDATE]: Term 2 election: Requested vote from peers a5d66df07715448fab64928f5bc8299a (127.16.94.130:39295), fceaf96557de46c3986c306fa4ac3b9e (127.16.94.131:39581)
I20240503 14:13:42.524235 20330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7905fd813a2d4a8aa8b2ac1e752b37dc" candidate_uuid: "64ad8799c8df4b7589869baffb93b68d" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "fceaf96557de46c3986c306fa4ac3b9e"
I20240503 14:13:42.524322 20261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7905fd813a2d4a8aa8b2ac1e752b37dc" candidate_uuid: "64ad8799c8df4b7589869baffb93b68d" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "a5d66df07715448fab64928f5bc8299a"
I20240503 14:13:42.524384 20330 raft_consensus.cc:3037] T 7905fd813a2d4a8aa8b2ac1e752b37dc P fceaf96557de46c3986c306fa4ac3b9e [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:42.524480 20261 raft_consensus.cc:3032] T 7905fd813a2d4a8aa8b2ac1e752b37dc P a5d66df07715448fab64928f5bc8299a [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:13:42.524530 20261 raft_consensus.cc:729] T 7905fd813a2d4a8aa8b2ac1e752b37dc P a5d66df07715448fab64928f5bc8299a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a5d66df07715448fab64928f5bc8299a, State: Running, Role: LEADER
I20240503 14:13:42.524802 20658 raft_consensus.cc:504] T 29f4ee03917646f6a5374217bed7a1be P c4d0f91a34804de99e3a2a255cd06aed [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "c4d0f91a34804de99e3a2a255cd06aed" member_type: VOTER last_known_addr { host: "127.16.94.132" port: 41945 } attrs { promote: false } }
I20240503 14:13:42.524763 20261 consensus_queue.cc:260] T 7905fd813a2d4a8aa8b2ac1e752b37dc P a5d66df07715448fab64928f5bc8299a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:42.525202 20261 raft_consensus.cc:3037] T 7905fd813a2d4a8aa8b2ac1e752b37dc P a5d66df07715448fab64928f5bc8299a [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:42.525954 20180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29f4ee03917646f6a5374217bed7a1be" candidate_uuid: "c4d0f91a34804de99e3a2a255cd06aed" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: true dest_uuid: "64ad8799c8df4b7589869baffb93b68d"
I20240503 14:13:42.526202 20180 raft_consensus.cc:3032] T 29f4ee03917646f6a5374217bed7a1be P 64ad8799c8df4b7589869baffb93b68d [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:13:42.526337 20658 leader_election.cc:288] T 29f4ee03917646f6a5374217bed7a1be P c4d0f91a34804de99e3a2a255cd06aed [CANDIDATE]: Term 2 election: Requested vote from peers 64ad8799c8df4b7589869baffb93b68d (127.16.94.129:37429), fceaf96557de46c3986c306fa4ac3b9e (127.16.94.131:39581)
I20240503 14:13:42.526335 20180 raft_consensus.cc:729] T 29f4ee03917646f6a5374217bed7a1be P 64ad8799c8df4b7589869baffb93b68d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 64ad8799c8df4b7589869baffb93b68d, State: Running, Role: LEADER
I20240503 14:13:42.527024 20330 raft_consensus.cc:2454] T 7905fd813a2d4a8aa8b2ac1e752b37dc P fceaf96557de46c3986c306fa4ac3b9e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64ad8799c8df4b7589869baffb93b68d in term 2.
I20240503 14:13:42.527499 20150 leader_election.cc:302] T 7905fd813a2d4a8aa8b2ac1e752b37dc P 64ad8799c8df4b7589869baffb93b68d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64ad8799c8df4b7589869baffb93b68d, fceaf96557de46c3986c306fa4ac3b9e; no voters:
I20240503 14:13:42.527845 20261 raft_consensus.cc:2454] T 7905fd813a2d4a8aa8b2ac1e752b37dc P a5d66df07715448fab64928f5bc8299a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64ad8799c8df4b7589869baffb93b68d in term 2.
I20240503 14:13:42.528041 20919 raft_consensus.cc:2781] T 7905fd813a2d4a8aa8b2ac1e752b37dc P 64ad8799c8df4b7589869baffb93b68d [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:13:42.528151 20919 raft_consensus.cc:686] T 7905fd813a2d4a8aa8b2ac1e752b37dc P 64ad8799c8df4b7589869baffb93b68d [term 2 LEADER]: Becoming Leader. State: Replica: 64ad8799c8df4b7589869baffb93b68d, State: Running, Role: LEADER
I20240503 14:13:42.528359 20919 consensus_queue.cc:237] T 7905fd813a2d4a8aa8b2ac1e752b37dc P 64ad8799c8df4b7589869baffb93b68d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:42.527359 20180 consensus_queue.cc:260] T 29f4ee03917646f6a5374217bed7a1be P 64ad8799c8df4b7589869baffb93b68d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "c4d0f91a34804de99e3a2a255cd06aed" member_type: VOTER last_known_addr { host: "127.16.94.132" port: 41945 } attrs { promote: false } }
I20240503 14:13:42.529280 20180 raft_consensus.cc:3037] T 29f4ee03917646f6a5374217bed7a1be P 64ad8799c8df4b7589869baffb93b68d [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:42.530961 20077 catalog_manager.cc:5484] T 7905fd813a2d4a8aa8b2ac1e752b37dc P 64ad8799c8df4b7589869baffb93b68d reported cstate change: term changed from 1 to 2, leader changed from a5d66df07715448fab64928f5bc8299a (127.16.94.130) to 64ad8799c8df4b7589869baffb93b68d (127.16.94.129). New cstate: current_term: 2 leader_uuid: "64ad8799c8df4b7589869baffb93b68d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:42.533537 20180 raft_consensus.cc:2454] T 29f4ee03917646f6a5374217bed7a1be P 64ad8799c8df4b7589869baffb93b68d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4d0f91a34804de99e3a2a255cd06aed in term 2.
I20240503 14:13:42.533994 20621 leader_election.cc:302] T 29f4ee03917646f6a5374217bed7a1be P c4d0f91a34804de99e3a2a255cd06aed [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64ad8799c8df4b7589869baffb93b68d, c4d0f91a34804de99e3a2a255cd06aed; no voters:
I20240503 14:13:42.533999 20330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29f4ee03917646f6a5374217bed7a1be" candidate_uuid: "c4d0f91a34804de99e3a2a255cd06aed" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: true dest_uuid: "fceaf96557de46c3986c306fa4ac3b9e"
I20240503 14:13:42.534215 20330 raft_consensus.cc:3037] T 29f4ee03917646f6a5374217bed7a1be P fceaf96557de46c3986c306fa4ac3b9e [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:42.534616 20968 raft_consensus.cc:2781] T 29f4ee03917646f6a5374217bed7a1be P c4d0f91a34804de99e3a2a255cd06aed [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:13:42.535046 20968 raft_consensus.cc:686] T 29f4ee03917646f6a5374217bed7a1be P c4d0f91a34804de99e3a2a255cd06aed [term 2 LEADER]: Becoming Leader. State: Replica: c4d0f91a34804de99e3a2a255cd06aed, State: Running, Role: LEADER
I20240503 14:13:42.535327 20968 consensus_queue.cc:237] T 29f4ee03917646f6a5374217bed7a1be P c4d0f91a34804de99e3a2a255cd06aed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "c4d0f91a34804de99e3a2a255cd06aed" member_type: VOTER last_known_addr { host: "127.16.94.132" port: 41945 } attrs { promote: false } }
I20240503 14:13:42.536988 20330 raft_consensus.cc:2454] T 29f4ee03917646f6a5374217bed7a1be P fceaf96557de46c3986c306fa4ac3b9e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4d0f91a34804de99e3a2a255cd06aed in term 2.
I20240503 14:13:42.538093 20078 catalog_manager.cc:5484] T 29f4ee03917646f6a5374217bed7a1be P c4d0f91a34804de99e3a2a255cd06aed reported cstate change: term changed from 1 to 2, leader changed from 64ad8799c8df4b7589869baffb93b68d (127.16.94.129) to c4d0f91a34804de99e3a2a255cd06aed (127.16.94.132). New cstate: current_term: 2 leader_uuid: "c4d0f91a34804de99e3a2a255cd06aed" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4d0f91a34804de99e3a2a255cd06aed" member_type: VOTER last_known_addr { host: "127.16.94.132" port: 41945 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:42.616817 20958 raft_consensus.cc:979] T c45ceab3a36d4225aed74654ed530c9d P a5d66df07715448fab64928f5bc8299a: : Instructing follower 64ad8799c8df4b7589869baffb93b68d to start an election
I20240503 14:13:42.616977 20958 raft_consensus.cc:1067] T c45ceab3a36d4225aed74654ed530c9d P a5d66df07715448fab64928f5bc8299a [term 1 LEADER]: Signalling peer 64ad8799c8df4b7589869baffb93b68d to start an election
I20240503 14:13:42.617475 20180 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c45ceab3a36d4225aed74654ed530c9d"
dest_uuid: "64ad8799c8df4b7589869baffb93b68d"
from {username='slave'} at 127.0.0.1:54502
I20240503 14:13:42.617628 20180 raft_consensus.cc:482] T c45ceab3a36d4225aed74654ed530c9d P 64ad8799c8df4b7589869baffb93b68d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:13:42.617717 20180 raft_consensus.cc:3037] T c45ceab3a36d4225aed74654ed530c9d P 64ad8799c8df4b7589869baffb93b68d [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:42.619469 20180 raft_consensus.cc:504] T c45ceab3a36d4225aed74654ed530c9d P 64ad8799c8df4b7589869baffb93b68d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:42.619932 20180 leader_election.cc:288] T c45ceab3a36d4225aed74654ed530c9d P 64ad8799c8df4b7589869baffb93b68d [CANDIDATE]: Term 2 election: Requested vote from peers a5d66df07715448fab64928f5bc8299a (127.16.94.130:39295), fceaf96557de46c3986c306fa4ac3b9e (127.16.94.131:39581)
I20240503 14:13:42.620276 20261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c45ceab3a36d4225aed74654ed530c9d" candidate_uuid: "64ad8799c8df4b7589869baffb93b68d" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "a5d66df07715448fab64928f5bc8299a"
I20240503 14:13:42.620290 20330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c45ceab3a36d4225aed74654ed530c9d" candidate_uuid: "64ad8799c8df4b7589869baffb93b68d" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "fceaf96557de46c3986c306fa4ac3b9e"
I20240503 14:13:42.620412 20261 raft_consensus.cc:3032] T c45ceab3a36d4225aed74654ed530c9d P a5d66df07715448fab64928f5bc8299a [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:13:42.620471 20261 raft_consensus.cc:729] T c45ceab3a36d4225aed74654ed530c9d P a5d66df07715448fab64928f5bc8299a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a5d66df07715448fab64928f5bc8299a, State: Running, Role: LEADER
I20240503 14:13:42.620496 20330 raft_consensus.cc:3037] T c45ceab3a36d4225aed74654ed530c9d P fceaf96557de46c3986c306fa4ac3b9e [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:42.620661 20261 consensus_queue.cc:260] T c45ceab3a36d4225aed74654ed530c9d P a5d66df07715448fab64928f5bc8299a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:42.621073 20261 raft_consensus.cc:3037] T c45ceab3a36d4225aed74654ed530c9d P a5d66df07715448fab64928f5bc8299a [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:42.622510 20330 raft_consensus.cc:2454] T c45ceab3a36d4225aed74654ed530c9d P fceaf96557de46c3986c306fa4ac3b9e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64ad8799c8df4b7589869baffb93b68d in term 2.
I20240503 14:13:42.622869 20150 leader_election.cc:302] T c45ceab3a36d4225aed74654ed530c9d P 64ad8799c8df4b7589869baffb93b68d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64ad8799c8df4b7589869baffb93b68d, fceaf96557de46c3986c306fa4ac3b9e; no voters:
I20240503 14:13:42.623152 20919 raft_consensus.cc:2781] T c45ceab3a36d4225aed74654ed530c9d P 64ad8799c8df4b7589869baffb93b68d [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:13:42.623234 20261 raft_consensus.cc:2454] T c45ceab3a36d4225aed74654ed530c9d P a5d66df07715448fab64928f5bc8299a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64ad8799c8df4b7589869baffb93b68d in term 2.
I20240503 14:13:42.623282 20919 raft_consensus.cc:686] T c45ceab3a36d4225aed74654ed530c9d P 64ad8799c8df4b7589869baffb93b68d [term 2 LEADER]: Becoming Leader. State: Replica: 64ad8799c8df4b7589869baffb93b68d, State: Running, Role: LEADER
I20240503 14:13:42.623500 20919 consensus_queue.cc:237] T c45ceab3a36d4225aed74654ed530c9d P 64ad8799c8df4b7589869baffb93b68d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:42.625437 20919 raft_consensus.cc:979] T 220ef8bab4694ace9bdf0fce06226efa P 64ad8799c8df4b7589869baffb93b68d: : Instructing follower f8c99fddf5974401b64aa8601605b2b2 to start an election
I20240503 14:13:42.625555 20919 raft_consensus.cc:1067] T 220ef8bab4694ace9bdf0fce06226efa P 64ad8799c8df4b7589869baffb93b68d [term 1 LEADER]: Signalling peer f8c99fddf5974401b64aa8601605b2b2 to start an election
I20240503 14:13:42.626101 20734 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "220ef8bab4694ace9bdf0fce06226efa"
dest_uuid: "f8c99fddf5974401b64aa8601605b2b2"
from {username='slave'} at 127.0.0.1:36776
I20240503 14:13:42.626272 20734 raft_consensus.cc:482] T 220ef8bab4694ace9bdf0fce06226efa P f8c99fddf5974401b64aa8601605b2b2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:13:42.626356 20734 raft_consensus.cc:3037] T 220ef8bab4694ace9bdf0fce06226efa P f8c99fddf5974401b64aa8601605b2b2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:42.626355 20078 catalog_manager.cc:5484] T c45ceab3a36d4225aed74654ed530c9d P 64ad8799c8df4b7589869baffb93b68d reported cstate change: term changed from 1 to 2, leader changed from a5d66df07715448fab64928f5bc8299a (127.16.94.130) to 64ad8799c8df4b7589869baffb93b68d (127.16.94.129). New cstate: current_term: 2 leader_uuid: "64ad8799c8df4b7589869baffb93b68d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:42.628435 20734 raft_consensus.cc:504] T 220ef8bab4694ace9bdf0fce06226efa P f8c99fddf5974401b64aa8601605b2b2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "f8c99fddf5974401b64aa8601605b2b2" member_type: VOTER last_known_addr { host: "127.16.94.133" port: 42559 } attrs { promote: false } }
I20240503 14:13:42.629158 20734 leader_election.cc:288] T 220ef8bab4694ace9bdf0fce06226efa P f8c99fddf5974401b64aa8601605b2b2 [CANDIDATE]: Term 2 election: Requested vote from peers 64ad8799c8df4b7589869baffb93b68d (127.16.94.129:37429), a5d66df07715448fab64928f5bc8299a (127.16.94.130:39295)
I20240503 14:13:42.629498 20180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "220ef8bab4694ace9bdf0fce06226efa" candidate_uuid: "f8c99fddf5974401b64aa8601605b2b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: true dest_uuid: "64ad8799c8df4b7589869baffb93b68d"
I20240503 14:13:42.629633 20180 raft_consensus.cc:3032] T 220ef8bab4694ace9bdf0fce06226efa P 64ad8799c8df4b7589869baffb93b68d [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:13:42.629722 20180 raft_consensus.cc:729] T 220ef8bab4694ace9bdf0fce06226efa P 64ad8799c8df4b7589869baffb93b68d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 64ad8799c8df4b7589869baffb93b68d, State: Running, Role: LEADER
I20240503 14:13:42.629954 20180 consensus_queue.cc:260] T 220ef8bab4694ace9bdf0fce06226efa P 64ad8799c8df4b7589869baffb93b68d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 30, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "f8c99fddf5974401b64aa8601605b2b2" member_type: VOTER last_known_addr { host: "127.16.94.133" port: 42559 } attrs { promote: false } }
I20240503 14:13:42.630201 20180 raft_consensus.cc:3037] T 220ef8bab4694ace9bdf0fce06226efa P 64ad8799c8df4b7589869baffb93b68d [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:42.631889 20180 raft_consensus.cc:2454] T 220ef8bab4694ace9bdf0fce06226efa P 64ad8799c8df4b7589869baffb93b68d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8c99fddf5974401b64aa8601605b2b2 in term 2.
I20240503 14:13:42.632229 20696 leader_election.cc:302] T 220ef8bab4694ace9bdf0fce06226efa P f8c99fddf5974401b64aa8601605b2b2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64ad8799c8df4b7589869baffb93b68d, f8c99fddf5974401b64aa8601605b2b2; no voters:
I20240503 14:13:42.632903 20975 raft_consensus.cc:2781] T 220ef8bab4694ace9bdf0fce06226efa P f8c99fddf5974401b64aa8601605b2b2 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:13:42.633335 20975 raft_consensus.cc:686] T 220ef8bab4694ace9bdf0fce06226efa P f8c99fddf5974401b64aa8601605b2b2 [term 2 LEADER]: Becoming Leader. State: Replica: f8c99fddf5974401b64aa8601605b2b2, State: Running, Role: LEADER
I20240503 14:13:42.633786 20975 consensus_queue.cc:237] T 220ef8bab4694ace9bdf0fce06226efa P f8c99fddf5974401b64aa8601605b2b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 30, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "f8c99fddf5974401b64aa8601605b2b2" member_type: VOTER last_known_addr { host: "127.16.94.133" port: 42559 } attrs { promote: false } }
I20240503 14:13:42.638549 20078 catalog_manager.cc:5484] T 220ef8bab4694ace9bdf0fce06226efa P f8c99fddf5974401b64aa8601605b2b2 reported cstate change: term changed from 1 to 2, leader changed from 64ad8799c8df4b7589869baffb93b68d (127.16.94.129) to f8c99fddf5974401b64aa8601605b2b2 (127.16.94.133). New cstate: current_term: 2 leader_uuid: "f8c99fddf5974401b64aa8601605b2b2" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8c99fddf5974401b64aa8601605b2b2" member_type: VOTER last_known_addr { host: "127.16.94.133" port: 42559 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:42.642474 20259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "220ef8bab4694ace9bdf0fce06226efa" candidate_uuid: "f8c99fddf5974401b64aa8601605b2b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: true dest_uuid: "a5d66df07715448fab64928f5bc8299a"
I20240503 14:13:42.642606 20259 raft_consensus.cc:3037] T 220ef8bab4694ace9bdf0fce06226efa P a5d66df07715448fab64928f5bc8299a [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:42.644382 20259 raft_consensus.cc:2454] T 220ef8bab4694ace9bdf0fce06226efa P a5d66df07715448fab64928f5bc8299a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8c99fddf5974401b64aa8601605b2b2 in term 2.
I20240503 14:13:42.695730 20916 raft_consensus.cc:979] T 0eb7552695304a1ea6f8e02f189f7295 P fceaf96557de46c3986c306fa4ac3b9e: : Instructing follower d6bcbc9fecfb44fa99a81917ea2d559a to start an election
I20240503 14:13:42.695871 20916 raft_consensus.cc:1067] T 0eb7552695304a1ea6f8e02f189f7295 P fceaf96557de46c3986c306fa4ac3b9e [term 1 LEADER]: Signalling peer d6bcbc9fecfb44fa99a81917ea2d559a to start an election
I20240503 14:13:42.696369 20807 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0eb7552695304a1ea6f8e02f189f7295"
dest_uuid: "d6bcbc9fecfb44fa99a81917ea2d559a"
from {username='slave'} at 127.0.0.1:44568
I20240503 14:13:42.696532 20807 raft_consensus.cc:482] T 0eb7552695304a1ea6f8e02f189f7295 P d6bcbc9fecfb44fa99a81917ea2d559a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:13:42.696615 20807 raft_consensus.cc:3037] T 0eb7552695304a1ea6f8e02f189f7295 P d6bcbc9fecfb44fa99a81917ea2d559a [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:42.698452 20807 raft_consensus.cc:504] T 0eb7552695304a1ea6f8e02f189f7295 P d6bcbc9fecfb44fa99a81917ea2d559a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "d6bcbc9fecfb44fa99a81917ea2d559a" member_type: VOTER last_known_addr { host: "127.16.94.134" port: 34983 } attrs { promote: false } }
I20240503 14:13:42.699087 20807 leader_election.cc:288] T 0eb7552695304a1ea6f8e02f189f7295 P d6bcbc9fecfb44fa99a81917ea2d559a [CANDIDATE]: Term 2 election: Requested vote from peers a5d66df07715448fab64928f5bc8299a (127.16.94.130:39295), fceaf96557de46c3986c306fa4ac3b9e (127.16.94.131:39581)
I20240503 14:13:42.699537 20330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eb7552695304a1ea6f8e02f189f7295" candidate_uuid: "d6bcbc9fecfb44fa99a81917ea2d559a" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: true dest_uuid: "fceaf96557de46c3986c306fa4ac3b9e"
I20240503 14:13:42.699731 20330 raft_consensus.cc:3032] T 0eb7552695304a1ea6f8e02f189f7295 P fceaf96557de46c3986c306fa4ac3b9e [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:13:42.699795 20330 raft_consensus.cc:729] T 0eb7552695304a1ea6f8e02f189f7295 P fceaf96557de46c3986c306fa4ac3b9e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fceaf96557de46c3986c306fa4ac3b9e, State: Running, Role: LEADER
I20240503 14:13:42.700024 20330 consensus_queue.cc:260] T 0eb7552695304a1ea6f8e02f189f7295 P fceaf96557de46c3986c306fa4ac3b9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "d6bcbc9fecfb44fa99a81917ea2d559a" member_type: VOTER last_known_addr { host: "127.16.94.134" port: 34983 } attrs { promote: false } }
I20240503 14:13:42.700300 20330 raft_consensus.cc:3037] T 0eb7552695304a1ea6f8e02f189f7295 P fceaf96557de46c3986c306fa4ac3b9e [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:42.702159 20330 raft_consensus.cc:2454] T 0eb7552695304a1ea6f8e02f189f7295 P fceaf96557de46c3986c306fa4ac3b9e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6bcbc9fecfb44fa99a81917ea2d559a in term 2.
I20240503 14:13:42.702469 20773 leader_election.cc:302] T 0eb7552695304a1ea6f8e02f189f7295 P d6bcbc9fecfb44fa99a81917ea2d559a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6bcbc9fecfb44fa99a81917ea2d559a, fceaf96557de46c3986c306fa4ac3b9e; no voters:
I20240503 14:13:42.703035 20980 raft_consensus.cc:2781] T 0eb7552695304a1ea6f8e02f189f7295 P d6bcbc9fecfb44fa99a81917ea2d559a [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:13:42.703388 20980 raft_consensus.cc:686] T 0eb7552695304a1ea6f8e02f189f7295 P d6bcbc9fecfb44fa99a81917ea2d559a [term 2 LEADER]: Becoming Leader. State: Replica: d6bcbc9fecfb44fa99a81917ea2d559a, State: Running, Role: LEADER
I20240503 14:13:42.703645 20980 consensus_queue.cc:237] T 0eb7552695304a1ea6f8e02f189f7295 P d6bcbc9fecfb44fa99a81917ea2d559a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "d6bcbc9fecfb44fa99a81917ea2d559a" member_type: VOTER last_known_addr { host: "127.16.94.134" port: 34983 } attrs { promote: false } }
I20240503 14:13:42.704259 20259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eb7552695304a1ea6f8e02f189f7295" candidate_uuid: "d6bcbc9fecfb44fa99a81917ea2d559a" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: true dest_uuid: "a5d66df07715448fab64928f5bc8299a"
I20240503 14:13:42.704391 20259 raft_consensus.cc:3037] T 0eb7552695304a1ea6f8e02f189f7295 P a5d66df07715448fab64928f5bc8299a [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:42.705935 20078 catalog_manager.cc:5484] T 0eb7552695304a1ea6f8e02f189f7295 P d6bcbc9fecfb44fa99a81917ea2d559a reported cstate change: term changed from 1 to 2, leader changed from fceaf96557de46c3986c306fa4ac3b9e (127.16.94.131) to d6bcbc9fecfb44fa99a81917ea2d559a (127.16.94.134). New cstate: current_term: 2 leader_uuid: "d6bcbc9fecfb44fa99a81917ea2d559a" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6bcbc9fecfb44fa99a81917ea2d559a" member_type: VOTER last_known_addr { host: "127.16.94.134" port: 34983 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:42.706352 20259 raft_consensus.cc:2454] T 0eb7552695304a1ea6f8e02f189f7295 P a5d66df07715448fab64928f5bc8299a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6bcbc9fecfb44fa99a81917ea2d559a in term 2.
I20240503 14:13:42.782225 20978 raft_consensus.cc:979] T ad711d78ee524d979867b65906abe456 P fceaf96557de46c3986c306fa4ac3b9e: : Instructing follower a5d66df07715448fab64928f5bc8299a to start an election
I20240503 14:13:42.782395 20978 raft_consensus.cc:1067] T ad711d78ee524d979867b65906abe456 P fceaf96557de46c3986c306fa4ac3b9e [term 1 LEADER]: Signalling peer a5d66df07715448fab64928f5bc8299a to start an election
I20240503 14:13:42.782900 20259 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ad711d78ee524d979867b65906abe456"
dest_uuid: "a5d66df07715448fab64928f5bc8299a"
from {username='slave'} at 127.0.0.1:38662
I20240503 14:13:42.783047 20259 raft_consensus.cc:482] T ad711d78ee524d979867b65906abe456 P a5d66df07715448fab64928f5bc8299a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:13:42.783110 20259 raft_consensus.cc:3037] T ad711d78ee524d979867b65906abe456 P a5d66df07715448fab64928f5bc8299a [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:42.784759 20259 raft_consensus.cc:504] T ad711d78ee524d979867b65906abe456 P a5d66df07715448fab64928f5bc8299a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:42.785230 20259 leader_election.cc:288] T ad711d78ee524d979867b65906abe456 P a5d66df07715448fab64928f5bc8299a [CANDIDATE]: Term 2 election: Requested vote from peers 64ad8799c8df4b7589869baffb93b68d (127.16.94.129:37429), fceaf96557de46c3986c306fa4ac3b9e (127.16.94.131:39581)
I20240503 14:13:42.785593 20330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad711d78ee524d979867b65906abe456" candidate_uuid: "a5d66df07715448fab64928f5bc8299a" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: true dest_uuid: "fceaf96557de46c3986c306fa4ac3b9e"
I20240503 14:13:42.785593 20180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad711d78ee524d979867b65906abe456" candidate_uuid: "a5d66df07715448fab64928f5bc8299a" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: true dest_uuid: "64ad8799c8df4b7589869baffb93b68d"
I20240503 14:13:42.785826 20330 raft_consensus.cc:3032] T ad711d78ee524d979867b65906abe456 P fceaf96557de46c3986c306fa4ac3b9e [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:13:42.785840 20180 raft_consensus.cc:3037] T ad711d78ee524d979867b65906abe456 P 64ad8799c8df4b7589869baffb93b68d [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:42.785914 20330 raft_consensus.cc:729] T ad711d78ee524d979867b65906abe456 P fceaf96557de46c3986c306fa4ac3b9e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fceaf96557de46c3986c306fa4ac3b9e, State: Running, Role: LEADER
I20240503 14:13:42.786151 20330 consensus_queue.cc:260] T ad711d78ee524d979867b65906abe456 P fceaf96557de46c3986c306fa4ac3b9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:42.786450 20330 raft_consensus.cc:3037] T ad711d78ee524d979867b65906abe456 P fceaf96557de46c3986c306fa4ac3b9e [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:42.788435 20180 raft_consensus.cc:2454] T ad711d78ee524d979867b65906abe456 P 64ad8799c8df4b7589869baffb93b68d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5d66df07715448fab64928f5bc8299a in term 2.
I20240503 14:13:42.788691 20330 raft_consensus.cc:2454] T ad711d78ee524d979867b65906abe456 P fceaf96557de46c3986c306fa4ac3b9e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5d66df07715448fab64928f5bc8299a in term 2.
I20240503 14:13:42.788806 20224 leader_election.cc:302] T ad711d78ee524d979867b65906abe456 P a5d66df07715448fab64928f5bc8299a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64ad8799c8df4b7589869baffb93b68d, a5d66df07715448fab64928f5bc8299a; no voters:
I20240503 14:13:42.789078 20958 raft_consensus.cc:2781] T ad711d78ee524d979867b65906abe456 P a5d66df07715448fab64928f5bc8299a [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:13:42.789189 20958 raft_consensus.cc:686] T ad711d78ee524d979867b65906abe456 P a5d66df07715448fab64928f5bc8299a [term 2 LEADER]: Becoming Leader. State: Replica: a5d66df07715448fab64928f5bc8299a, State: Running, Role: LEADER
I20240503 14:13:42.789415 20958 consensus_queue.cc:237] T ad711d78ee524d979867b65906abe456 P a5d66df07715448fab64928f5bc8299a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } }
I20240503 14:13:42.791760 20078 catalog_manager.cc:5484] T ad711d78ee524d979867b65906abe456 P a5d66df07715448fab64928f5bc8299a reported cstate change: term changed from 1 to 2, leader changed from fceaf96557de46c3986c306fa4ac3b9e (127.16.94.131) to a5d66df07715448fab64928f5bc8299a (127.16.94.130). New cstate: current_term: 2 leader_uuid: "a5d66df07715448fab64928f5bc8299a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:42.977180 20259 raft_consensus.cc:1261] T 7905fd813a2d4a8aa8b2ac1e752b37dc P a5d66df07715448fab64928f5bc8299a [term 2 FOLLOWER]: Refusing update from remote peer 64ad8799c8df4b7589869baffb93b68d: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240503 14:13:42.977778 20919 consensus_queue.cc:1035] T 7905fd813a2d4a8aa8b2ac1e752b37dc P 64ad8799c8df4b7589869baffb93b68d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 14:13:42.982208 20330 raft_consensus.cc:1261] T 7905fd813a2d4a8aa8b2ac1e752b37dc P fceaf96557de46c3986c306fa4ac3b9e [term 2 FOLLOWER]: Refusing update from remote peer 64ad8799c8df4b7589869baffb93b68d: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240503 14:13:42.982841 20919 consensus_queue.cc:1035] T 7905fd813a2d4a8aa8b2ac1e752b37dc P 64ad8799c8df4b7589869baffb93b68d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 14:13:43.019579 20330 raft_consensus.cc:1261] T c45ceab3a36d4225aed74654ed530c9d P fceaf96557de46c3986c306fa4ac3b9e [term 2 FOLLOWER]: Refusing update from remote peer 64ad8799c8df4b7589869baffb93b68d: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240503 14:13:43.020131 20919 consensus_queue.cc:1035] T c45ceab3a36d4225aed74654ed530c9d P 64ad8799c8df4b7589869baffb93b68d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 14:13:43.024036 20259 raft_consensus.cc:1261] T c45ceab3a36d4225aed74654ed530c9d P a5d66df07715448fab64928f5bc8299a [term 2 FOLLOWER]: Refusing update from remote peer 64ad8799c8df4b7589869baffb93b68d: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240503 14:13:43.024484 20987 consensus_queue.cc:1035] T c45ceab3a36d4225aed74654ed530c9d P 64ad8799c8df4b7589869baffb93b68d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 14:13:43.059728 20180 raft_consensus.cc:1261] T 29f4ee03917646f6a5374217bed7a1be P 64ad8799c8df4b7589869baffb93b68d [term 2 FOLLOWER]: Refusing update from remote peer c4d0f91a34804de99e3a2a255cd06aed: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240503 14:13:43.060223 20992 consensus_queue.cc:1035] T 29f4ee03917646f6a5374217bed7a1be P c4d0f91a34804de99e3a2a255cd06aed [LEADER]: Connected to new peer: Peer: permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 14:13:43.063726 20330 raft_consensus.cc:1261] T 29f4ee03917646f6a5374217bed7a1be P fceaf96557de46c3986c306fa4ac3b9e [term 2 FOLLOWER]: Refusing update from remote peer c4d0f91a34804de99e3a2a255cd06aed: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240503 14:13:43.064358 20994 consensus_queue.cc:1035] T 29f4ee03917646f6a5374217bed7a1be P c4d0f91a34804de99e3a2a255cd06aed [LEADER]: Connected to new peer: Peer: permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 14:13:43.079522 20180 raft_consensus.cc:1261] T 220ef8bab4694ace9bdf0fce06226efa P 64ad8799c8df4b7589869baffb93b68d [term 2 FOLLOWER]: Refusing update from remote peer f8c99fddf5974401b64aa8601605b2b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240503 14:13:43.079972 20975 consensus_queue.cc:1035] T 220ef8bab4694ace9bdf0fce06226efa P f8c99fddf5974401b64aa8601605b2b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 14:13:43.083333 20259 raft_consensus.cc:1261] T 220ef8bab4694ace9bdf0fce06226efa P a5d66df07715448fab64928f5bc8299a [term 2 FOLLOWER]: Refusing update from remote peer f8c99fddf5974401b64aa8601605b2b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240503 14:13:43.083838 21001 consensus_queue.cc:1035] T 220ef8bab4694ace9bdf0fce06226efa P f8c99fddf5974401b64aa8601605b2b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 14:13:43.167586 20259 raft_consensus.cc:1261] T 0eb7552695304a1ea6f8e02f189f7295 P a5d66df07715448fab64928f5bc8299a [term 2 FOLLOWER]: Refusing update from remote peer d6bcbc9fecfb44fa99a81917ea2d559a: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240503 14:13:43.168118 20980 consensus_queue.cc:1035] T 0eb7552695304a1ea6f8e02f189f7295 P d6bcbc9fecfb44fa99a81917ea2d559a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 14:13:43.171751 20330 raft_consensus.cc:1261] T 0eb7552695304a1ea6f8e02f189f7295 P fceaf96557de46c3986c306fa4ac3b9e [term 2 FOLLOWER]: Refusing update from remote peer d6bcbc9fecfb44fa99a81917ea2d559a: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240503 14:13:43.172240 20980 consensus_queue.cc:1035] T 0eb7552695304a1ea6f8e02f189f7295 P d6bcbc9fecfb44fa99a81917ea2d559a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 14:13:43.285959 20180 raft_consensus.cc:1261] T ad711d78ee524d979867b65906abe456 P 64ad8799c8df4b7589869baffb93b68d [term 2 FOLLOWER]: Refusing update from remote peer a5d66df07715448fab64928f5bc8299a: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20240503 14:13:43.286499 20958 consensus_queue.cc:1035] T ad711d78ee524d979867b65906abe456 P a5d66df07715448fab64928f5bc8299a [LEADER]: Connected to new peer: Peer: permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240503 14:13:43.289484 20330 raft_consensus.cc:1261] T ad711d78ee524d979867b65906abe456 P fceaf96557de46c3986c306fa4ac3b9e [term 2 FOLLOWER]: Refusing update from remote peer a5d66df07715448fab64928f5bc8299a: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20240503 14:13:43.290194 21007 consensus_queue.cc:1035] T ad711d78ee524d979867b65906abe456 P a5d66df07715448fab64928f5bc8299a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240503 14:13:43.462419 20259 consensus_queue.cc:237] T 44a4d2b820fc4d0da06c659fe98fe57c P a5d66df07715448fab64928f5bc8299a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 3.27, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } attrs { replace: true } } peers { permanent_uuid: "d6bcbc9fecfb44fa99a81917ea2d559a" member_type: NON_VOTER last_known_addr { host: "127.16.94.134" port: 34983 } attrs { promote: true } }
I20240503 14:13:43.464859 20180 raft_consensus.cc:1261] T 44a4d2b820fc4d0da06c659fe98fe57c P 64ad8799c8df4b7589869baffb93b68d [term 3 FOLLOWER]: Refusing update from remote peer a5d66df07715448fab64928f5bc8299a: Log matching property violated. Preceding OpId in replica: term: 3 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20240503 14:13:43.465137 20330 raft_consensus.cc:1261] T 44a4d2b820fc4d0da06c659fe98fe57c P fceaf96557de46c3986c306fa4ac3b9e [term 3 FOLLOWER]: Refusing update from remote peer a5d66df07715448fab64928f5bc8299a: Log matching property violated. Preceding OpId in replica: term: 3 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20240503 14:13:43.465435 20958 consensus_queue.cc:1035] T 44a4d2b820fc4d0da06c659fe98fe57c P a5d66df07715448fab64928f5bc8299a [LEADER]: Connected to new peer: Peer: permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 14:13:43.466312 20958 consensus_queue.cc:1035] T 44a4d2b820fc4d0da06c659fe98fe57c P a5d66df07715448fab64928f5bc8299a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 14:13:43.468295 21007 raft_consensus.cc:2932] T 44a4d2b820fc4d0da06c659fe98fe57c P a5d66df07715448fab64928f5bc8299a [term 3 LEADER]: Committing config change with OpId 3.28: config changed from index -1 to 28, NON_VOTER d6bcbc9fecfb44fa99a81917ea2d559a (127.16.94.134) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } attrs { replace: true } } peers { permanent_uuid: "d6bcbc9fecfb44fa99a81917ea2d559a" member_type: NON_VOTER last_known_addr { host: "127.16.94.134" port: 34983 } attrs { promote: true } } }
I20240503 14:13:43.468899 20180 raft_consensus.cc:2932] T 44a4d2b820fc4d0da06c659fe98fe57c P 64ad8799c8df4b7589869baffb93b68d [term 3 FOLLOWER]: Committing config change with OpId 3.28: config changed from index -1 to 28, NON_VOTER d6bcbc9fecfb44fa99a81917ea2d559a (127.16.94.134) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } attrs { replace: true } } peers { permanent_uuid: "d6bcbc9fecfb44fa99a81917ea2d559a" member_type: NON_VOTER last_known_addr { host: "127.16.94.134" port: 34983 } attrs { promote: true } } }
I20240503 14:13:43.470621 20330 raft_consensus.cc:2932] T 44a4d2b820fc4d0da06c659fe98fe57c P fceaf96557de46c3986c306fa4ac3b9e [term 3 FOLLOWER]: Committing config change with OpId 3.28: config changed from index -1 to 28, NON_VOTER d6bcbc9fecfb44fa99a81917ea2d559a (127.16.94.134) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } attrs { replace: true } } peers { permanent_uuid: "d6bcbc9fecfb44fa99a81917ea2d559a" member_type: NON_VOTER last_known_addr { host: "127.16.94.134" port: 34983 } attrs { promote: true } } }
W20240503 14:13:43.472329 20225 consensus_peers.cc:485] T 44a4d2b820fc4d0da06c659fe98fe57c P a5d66df07715448fab64928f5bc8299a -> Peer d6bcbc9fecfb44fa99a81917ea2d559a (127.16.94.134:34983): Couldn't send request to peer d6bcbc9fecfb44fa99a81917ea2d559a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 44a4d2b820fc4d0da06c659fe98fe57c. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:43.474520 20259 consensus_queue.cc:237] T ad711d78ee524d979867b65906abe456 P a5d66df07715448fab64928f5bc8299a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 2.26, Last appended by leader: 25, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } attrs { replace: true } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "c4d0f91a34804de99e3a2a255cd06aed" member_type: NON_VOTER last_known_addr { host: "127.16.94.132" port: 41945 } attrs { promote: true } }
I20240503 14:13:43.474850 20077 catalog_manager.cc:5484] T 44a4d2b820fc4d0da06c659fe98fe57c P 64ad8799c8df4b7589869baffb93b68d reported cstate change: config changed from index -1 to 28, NON_VOTER d6bcbc9fecfb44fa99a81917ea2d559a (127.16.94.134) added. New cstate: current_term: 3 leader_uuid: "a5d66df07715448fab64928f5bc8299a" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } attrs { replace: true } } peers { permanent_uuid: "d6bcbc9fecfb44fa99a81917ea2d559a" member_type: NON_VOTER last_known_addr { host: "127.16.94.134" port: 34983 } attrs { promote: true } } }
W20240503 14:13:43.476833 20226 consensus_peers.cc:485] T ad711d78ee524d979867b65906abe456 P a5d66df07715448fab64928f5bc8299a -> Peer c4d0f91a34804de99e3a2a255cd06aed (127.16.94.132:41945): Couldn't send request to peer c4d0f91a34804de99e3a2a255cd06aed. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ad711d78ee524d979867b65906abe456. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:43.477185 20330 raft_consensus.cc:1261] T ad711d78ee524d979867b65906abe456 P fceaf96557de46c3986c306fa4ac3b9e [term 2 FOLLOWER]: Refusing update from remote peer a5d66df07715448fab64928f5bc8299a: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240503 14:13:43.477502 20180 raft_consensus.cc:1261] T ad711d78ee524d979867b65906abe456 P 64ad8799c8df4b7589869baffb93b68d [term 2 FOLLOWER]: Refusing update from remote peer a5d66df07715448fab64928f5bc8299a: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240503 14:13:43.477772 20932 consensus_queue.cc:1035] T ad711d78ee524d979867b65906abe456 P a5d66df07715448fab64928f5bc8299a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 14:13:43.478199 20932 consensus_queue.cc:1035] T ad711d78ee524d979867b65906abe456 P a5d66df07715448fab64928f5bc8299a [LEADER]: Connected to new peer: Peer: permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 14:13:43.479892 21007 raft_consensus.cc:2932] T ad711d78ee524d979867b65906abe456 P a5d66df07715448fab64928f5bc8299a [term 2 LEADER]: Committing config change with OpId 2.27: config changed from index -1 to 27, NON_VOTER c4d0f91a34804de99e3a2a255cd06aed (127.16.94.132) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } attrs { replace: true } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "c4d0f91a34804de99e3a2a255cd06aed" member_type: NON_VOTER last_known_addr { host: "127.16.94.132" port: 41945 } attrs { promote: true } } }
I20240503 14:13:43.481468 20330 raft_consensus.cc:2932] T ad711d78ee524d979867b65906abe456 P fceaf96557de46c3986c306fa4ac3b9e [term 2 FOLLOWER]: Committing config change with OpId 2.27: config changed from index -1 to 27, NON_VOTER c4d0f91a34804de99e3a2a255cd06aed (127.16.94.132) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } attrs { replace: true } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "c4d0f91a34804de99e3a2a255cd06aed" member_type: NON_VOTER last_known_addr { host: "127.16.94.132" port: 41945 } attrs { promote: true } } }
I20240503 14:13:43.482015 20180 raft_consensus.cc:2932] T ad711d78ee524d979867b65906abe456 P 64ad8799c8df4b7589869baffb93b68d [term 2 FOLLOWER]: Committing config change with OpId 2.27: config changed from index -1 to 27, NON_VOTER c4d0f91a34804de99e3a2a255cd06aed (127.16.94.132) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } attrs { replace: true } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "c4d0f91a34804de99e3a2a255cd06aed" member_type: NON_VOTER last_known_addr { host: "127.16.94.132" port: 41945 } attrs { promote: true } } }
I20240503 14:13:43.486016 20179 consensus_queue.cc:237] T 7a82ae1f4c66455caea352a304af2b51 P 64ad8799c8df4b7589869baffb93b68d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } attrs { replace: true } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "f8c99fddf5974401b64aa8601605b2b2" member_type: NON_VOTER last_known_addr { host: "127.16.94.133" port: 42559 } attrs { promote: true } }
I20240503 14:13:43.486624 20078 catalog_manager.cc:5484] T ad711d78ee524d979867b65906abe456 P a5d66df07715448fab64928f5bc8299a reported cstate change: config changed from index -1 to 27, NON_VOTER c4d0f91a34804de99e3a2a255cd06aed (127.16.94.132) added. New cstate: current_term: 2 leader_uuid: "a5d66df07715448fab64928f5bc8299a" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4d0f91a34804de99e3a2a255cd06aed" member_type: NON_VOTER last_known_addr { host: "127.16.94.132" port: 41945 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:43.488431 20259 raft_consensus.cc:1261] T 7a82ae1f4c66455caea352a304af2b51 P a5d66df07715448fab64928f5bc8299a [term 2 FOLLOWER]: Refusing update from remote peer 64ad8799c8df4b7589869baffb93b68d: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240503 14:13:43.488869 20330 raft_consensus.cc:1261] T 7a82ae1f4c66455caea352a304af2b51 P fceaf96557de46c3986c306fa4ac3b9e [term 2 FOLLOWER]: Refusing update from remote peer 64ad8799c8df4b7589869baffb93b68d: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
W20240503 14:13:43.488875 20151 consensus_peers.cc:485] T 7a82ae1f4c66455caea352a304af2b51 P 64ad8799c8df4b7589869baffb93b68d -> Peer f8c99fddf5974401b64aa8601605b2b2 (127.16.94.133:42559): Couldn't send request to peer f8c99fddf5974401b64aa8601605b2b2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7a82ae1f4c66455caea352a304af2b51. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:43.489415 20987 consensus_queue.cc:1035] T 7a82ae1f4c66455caea352a304af2b51 P 64ad8799c8df4b7589869baffb93b68d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 14:13:43.489763 20919 consensus_queue.cc:1035] T 7a82ae1f4c66455caea352a304af2b51 P 64ad8799c8df4b7589869baffb93b68d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 14:13:43.494052 20987 raft_consensus.cc:2932] T 7a82ae1f4c66455caea352a304af2b51 P 64ad8799c8df4b7589869baffb93b68d [term 2 LEADER]: Committing config change with OpId 2.29: config changed from index -1 to 29, NON_VOTER f8c99fddf5974401b64aa8601605b2b2 (127.16.94.133) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } attrs { replace: true } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "f8c99fddf5974401b64aa8601605b2b2" member_type: NON_VOTER last_known_addr { host: "127.16.94.133" port: 42559 } attrs { promote: true } } }
I20240503 14:13:43.494724 20329 raft_consensus.cc:2932] T 7a82ae1f4c66455caea352a304af2b51 P fceaf96557de46c3986c306fa4ac3b9e [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index -1 to 29, NON_VOTER f8c99fddf5974401b64aa8601605b2b2 (127.16.94.133) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } attrs { replace: true } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "f8c99fddf5974401b64aa8601605b2b2" member_type: NON_VOTER last_known_addr { host: "127.16.94.133" port: 42559 } attrs { promote: true } } }
I20240503 14:13:43.495275 20259 raft_consensus.cc:2932] T 7a82ae1f4c66455caea352a304af2b51 P a5d66df07715448fab64928f5bc8299a [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index -1 to 29, NON_VOTER f8c99fddf5974401b64aa8601605b2b2 (127.16.94.133) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } attrs { replace: true } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "f8c99fddf5974401b64aa8601605b2b2" member_type: NON_VOTER last_known_addr { host: "127.16.94.133" port: 42559 } attrs { promote: true } } }
I20240503 14:13:43.499574 20081 catalog_manager.cc:5484] T 7a82ae1f4c66455caea352a304af2b51 P 64ad8799c8df4b7589869baffb93b68d reported cstate change: config changed from index -1 to 29, NON_VOTER f8c99fddf5974401b64aa8601605b2b2 (127.16.94.133) added. New cstate: current_term: 2 leader_uuid: "64ad8799c8df4b7589869baffb93b68d" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8c99fddf5974401b64aa8601605b2b2" member_type: NON_VOTER last_known_addr { host: "127.16.94.133" port: 42559 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:43.500033 20179 consensus_queue.cc:237] T c45ceab3a36d4225aed74654ed530c9d P 64ad8799c8df4b7589869baffb93b68d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 27, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } attrs { replace: true } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "c4d0f91a34804de99e3a2a255cd06aed" member_type: NON_VOTER last_known_addr { host: "127.16.94.132" port: 41945 } attrs { promote: true } }
W20240503 14:13:43.502708 20150 consensus_peers.cc:485] T c45ceab3a36d4225aed74654ed530c9d P 64ad8799c8df4b7589869baffb93b68d -> Peer c4d0f91a34804de99e3a2a255cd06aed (127.16.94.132:41945): Couldn't send request to peer c4d0f91a34804de99e3a2a255cd06aed. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c45ceab3a36d4225aed74654ed530c9d. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:43.503336 20259 raft_consensus.cc:1261] T c45ceab3a36d4225aed74654ed530c9d P a5d66df07715448fab64928f5bc8299a [term 2 FOLLOWER]: Refusing update from remote peer 64ad8799c8df4b7589869baffb93b68d: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240503 14:13:43.504140 20329 raft_consensus.cc:1261] T c45ceab3a36d4225aed74654ed530c9d P fceaf96557de46c3986c306fa4ac3b9e [term 2 FOLLOWER]: Refusing update from remote peer 64ad8799c8df4b7589869baffb93b68d: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240503 14:13:43.504518 20987 consensus_queue.cc:1035] T c45ceab3a36d4225aed74654ed530c9d P 64ad8799c8df4b7589869baffb93b68d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 14:13:43.504815 20919 consensus_queue.cc:1035] T c45ceab3a36d4225aed74654ed530c9d P 64ad8799c8df4b7589869baffb93b68d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 14:13:43.506372 20987 raft_consensus.cc:2932] T c45ceab3a36d4225aed74654ed530c9d P 64ad8799c8df4b7589869baffb93b68d [term 2 LEADER]: Committing config change with OpId 2.29: config changed from index -1 to 29, NON_VOTER c4d0f91a34804de99e3a2a255cd06aed (127.16.94.132) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } attrs { replace: true } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "c4d0f91a34804de99e3a2a255cd06aed" member_type: NON_VOTER last_known_addr { host: "127.16.94.132" port: 41945 } attrs { promote: true } } }
I20240503 14:13:43.507336 20259 raft_consensus.cc:2932] T c45ceab3a36d4225aed74654ed530c9d P a5d66df07715448fab64928f5bc8299a [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index -1 to 29, NON_VOTER c4d0f91a34804de99e3a2a255cd06aed (127.16.94.132) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } attrs { replace: true } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "c4d0f91a34804de99e3a2a255cd06aed" member_type: NON_VOTER last_known_addr { host: "127.16.94.132" port: 41945 } attrs { promote: true } } }
I20240503 14:13:43.507755 20329 raft_consensus.cc:2932] T c45ceab3a36d4225aed74654ed530c9d P fceaf96557de46c3986c306fa4ac3b9e [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index -1 to 29, NON_VOTER c4d0f91a34804de99e3a2a255cd06aed (127.16.94.132) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } attrs { replace: true } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "c4d0f91a34804de99e3a2a255cd06aed" member_type: NON_VOTER last_known_addr { host: "127.16.94.132" port: 41945 } attrs { promote: true } } }
I20240503 14:13:43.513339 20078 catalog_manager.cc:5484] T c45ceab3a36d4225aed74654ed530c9d P a5d66df07715448fab64928f5bc8299a reported cstate change: config changed from index -1 to 29, NON_VOTER c4d0f91a34804de99e3a2a255cd06aed (127.16.94.132) added. New cstate: current_term: 2 leader_uuid: "64ad8799c8df4b7589869baffb93b68d" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } } peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } attrs { replace: true } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "c4d0f91a34804de99e3a2a255cd06aed" member_type: NON_VOTER last_known_addr { host: "127.16.94.132" port: 41945 } attrs { promote: true } } }
I20240503 14:13:43.528260 20658 consensus_queue.cc:237] T 29f4ee03917646f6a5374217bed7a1be P c4d0f91a34804de99e3a2a255cd06aed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 29, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } attrs { replace: true } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "c4d0f91a34804de99e3a2a255cd06aed" member_type: VOTER last_known_addr { host: "127.16.94.132" port: 41945 } attrs { promote: false } } peers { permanent_uuid: "d6bcbc9fecfb44fa99a81917ea2d559a" member_type: NON_VOTER last_known_addr { host: "127.16.94.134" port: 34983 } attrs { promote: true } }
I20240503 14:13:43.530335 20180 raft_consensus.cc:1261] T 29f4ee03917646f6a5374217bed7a1be P 64ad8799c8df4b7589869baffb93b68d [term 2 FOLLOWER]: Refusing update from remote peer c4d0f91a34804de99e3a2a255cd06aed: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240503 14:13:43.530462 20329 raft_consensus.cc:1261] T 29f4ee03917646f6a5374217bed7a1be P fceaf96557de46c3986c306fa4ac3b9e [term 2 FOLLOWER]: Refusing update from remote peer c4d0f91a34804de99e3a2a255cd06aed: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240503 14:13:43.530844 20996 consensus_queue.cc:1035] T 29f4ee03917646f6a5374217bed7a1be P c4d0f91a34804de99e3a2a255cd06aed [LEADER]: Connected to new peer: Peer: permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 14:13:43.531086 20997 consensus_queue.cc:1035] T 29f4ee03917646f6a5374217bed7a1be P c4d0f91a34804de99e3a2a255cd06aed [LEADER]: Connected to new peer: Peer: permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 14:13:43.533027 20996 raft_consensus.cc:2932] T 29f4ee03917646f6a5374217bed7a1be P c4d0f91a34804de99e3a2a255cd06aed [term 2 LEADER]: Committing config change with OpId 2.31: config changed from index 29 to 31, NON_VOTER d6bcbc9fecfb44fa99a81917ea2d559a (127.16.94.134) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } attrs { replace: true } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "c4d0f91a34804de99e3a2a255cd06aed" member_type: VOTER last_known_addr { host: "127.16.94.132" port: 41945 } attrs { promote: false } } peers { permanent_uuid: "d6bcbc9fecfb44fa99a81917ea2d559a" member_type: NON_VOTER last_known_addr { host: "127.16.94.134" port: 34983 } attrs { promote: true } } }
I20240503 14:13:43.533700 20180 raft_consensus.cc:2932] T 29f4ee03917646f6a5374217bed7a1be P 64ad8799c8df4b7589869baffb93b68d [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 29 to 31, NON_VOTER d6bcbc9fecfb44fa99a81917ea2d559a (127.16.94.134) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } attrs { replace: true } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "c4d0f91a34804de99e3a2a255cd06aed" member_type: VOTER last_known_addr { host: "127.16.94.132" port: 41945 } attrs { promote: false } } peers { permanent_uuid: "d6bcbc9fecfb44fa99a81917ea2d559a" member_type: NON_VOTER last_known_addr { host: "127.16.94.134" port: 34983 } attrs { promote: true } } }
I20240503 14:13:43.534689 20329 raft_consensus.cc:2932] T 29f4ee03917646f6a5374217bed7a1be P fceaf96557de46c3986c306fa4ac3b9e [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 29 to 31, NON_VOTER d6bcbc9fecfb44fa99a81917ea2d559a (127.16.94.134) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } attrs { replace: true } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "c4d0f91a34804de99e3a2a255cd06aed" member_type: VOTER last_known_addr { host: "127.16.94.132" port: 41945 } attrs { promote: false } } peers { permanent_uuid: "d6bcbc9fecfb44fa99a81917ea2d559a" member_type: NON_VOTER last_known_addr { host: "127.16.94.134" port: 34983 } attrs { promote: true } } }
I20240503 14:13:43.538914 20081 catalog_manager.cc:5484] T 29f4ee03917646f6a5374217bed7a1be P 64ad8799c8df4b7589869baffb93b68d reported cstate change: config changed from index 29 to 31, NON_VOTER d6bcbc9fecfb44fa99a81917ea2d559a (127.16.94.134) added. New cstate: current_term: 2 leader_uuid: "c4d0f91a34804de99e3a2a255cd06aed" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "64ad8799c8df4b7589869baffb93b68d" member_type: VOTER last_known_addr { host: "127.16.94.129" port: 37429 } attrs { replace: true } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } } peers { permanent_uuid: "c4d0f91a34804de99e3a2a255cd06aed" member_type: VOTER last_known_addr { host: "127.16.94.132" port: 41945 } attrs { promote: false } } peers { permanent_uuid: "d6bcbc9fecfb44fa99a81917ea2d559a" member_type: NON_VOTER last_known_addr { host: "127.16.94.134" port: 34983 } attrs { promote: true } } }
W20240503 14:13:43.551718 20622 consensus_peers.cc:485] T 29f4ee03917646f6a5374217bed7a1be P c4d0f91a34804de99e3a2a255cd06aed -> Peer d6bcbc9fecfb44fa99a81917ea2d559a (127.16.94.134:34983): Couldn't send request to peer d6bcbc9fecfb44fa99a81917ea2d559a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 29f4ee03917646f6a5374217bed7a1be. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:43.552722 20807 consensus_queue.cc:237] T 0eb7552695304a1ea6f8e02f189f7295 P d6bcbc9fecfb44fa99a81917ea2d559a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 29, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } attrs { replace: true } } peers { permanent_uuid: "d6bcbc9fecfb44fa99a81917ea2d559a" member_type: VOTER last_known_addr { host: "127.16.94.134" port: 34983 } attrs { promote: false } } peers { permanent_uuid: "f8c99fddf5974401b64aa8601605b2b2" member_type: NON_VOTER last_known_addr { host: "127.16.94.133" port: 42559 } attrs { promote: true } }
I20240503 14:13:43.555254 20259 raft_consensus.cc:1261] T 0eb7552695304a1ea6f8e02f189f7295 P a5d66df07715448fab64928f5bc8299a [term 2 FOLLOWER]: Refusing update from remote peer d6bcbc9fecfb44fa99a81917ea2d559a: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240503 14:13:43.555788 20980 consensus_queue.cc:1035] T 0eb7552695304a1ea6f8e02f189f7295 P d6bcbc9fecfb44fa99a81917ea2d559a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 14:13:43.555966 20329 raft_consensus.cc:1261] T 0eb7552695304a1ea6f8e02f189f7295 P fceaf96557de46c3986c306fa4ac3b9e [term 2 FOLLOWER]: Refusing update from remote peer d6bcbc9fecfb44fa99a81917ea2d559a: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240503 14:13:43.556785 21025 consensus_queue.cc:1035] T 0eb7552695304a1ea6f8e02f189f7295 P d6bcbc9fecfb44fa99a81917ea2d559a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 14:13:43.558815 20259 raft_consensus.cc:2932] T 0eb7552695304a1ea6f8e02f189f7295 P a5d66df07715448fab64928f5bc8299a [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 29 to 31, NON_VOTER f8c99fddf5974401b64aa8601605b2b2 (127.16.94.133) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } attrs { replace: true } } peers { permanent_uuid: "d6bcbc9fecfb44fa99a81917ea2d559a" member_type: VOTER last_known_addr { host: "127.16.94.134" port: 34983 } attrs { promote: false } } peers { permanent_uuid: "f8c99fddf5974401b64aa8601605b2b2" member_type: NON_VOTER last_known_addr { host: "127.16.94.133" port: 42559 } attrs { promote: true } } }
I20240503 14:13:43.558188 21025 raft_consensus.cc:2932] T 0eb7552695304a1ea6f8e02f189f7295 P d6bcbc9fecfb44fa99a81917ea2d559a [term 2 LEADER]: Committing config change with OpId 2.31: config changed from index 29 to 31, NON_VOTER f8c99fddf5974401b64aa8601605b2b2 (127.16.94.133) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } attrs { replace: true } } peers { permanent_uuid: "d6bcbc9fecfb44fa99a81917ea2d559a" member_type: VOTER last_known_addr { host: "127.16.94.134" port: 34983 } attrs { promote: false } } peers { permanent_uuid: "f8c99fddf5974401b64aa8601605b2b2" member_type: NON_VOTER last_known_addr { host: "127.16.94.133" port: 42559 } attrs { promote: true } } }
I20240503 14:13:43.559686 20329 raft_consensus.cc:2932] T 0eb7552695304a1ea6f8e02f189f7295 P fceaf96557de46c3986c306fa4ac3b9e [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 29 to 31, NON_VOTER f8c99fddf5974401b64aa8601605b2b2 (127.16.94.133) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } attrs { replace: true } } peers { permanent_uuid: "d6bcbc9fecfb44fa99a81917ea2d559a" member_type: VOTER last_known_addr { host: "127.16.94.134" port: 34983 } attrs { promote: false } } peers { permanent_uuid: "f8c99fddf5974401b64aa8601605b2b2" member_type: NON_VOTER last_known_addr { host: "127.16.94.133" port: 42559 } attrs { promote: true } } }
I20240503 14:13:43.564874 20078 catalog_manager.cc:5484] T 0eb7552695304a1ea6f8e02f189f7295 P a5d66df07715448fab64928f5bc8299a reported cstate change: config changed from index 29 to 31, NON_VOTER f8c99fddf5974401b64aa8601605b2b2 (127.16.94.133) added. New cstate: current_term: 2 leader_uuid: "d6bcbc9fecfb44fa99a81917ea2d559a" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "a5d66df07715448fab64928f5bc8299a" member_type: VOTER last_known_addr { host: "127.16.94.130" port: 39295 } } peers { permanent_uuid: "fceaf96557de46c3986c306fa4ac3b9e" member_type: VOTER last_known_addr { host: "127.16.94.131" port: 39581 } attrs { replace: true } } peers { permanent_uuid: "d6bcbc9fecfb44fa99a81917ea2d559a" member_type: VOTER last_known_addr { host: "127.16.94.134" port: 34983 } attrs { promote: false } } peers { permanent_uuid: "f8c99fddf5974401b64aa8601605b2b2" member_type: NON_VOTER last_known_addr { host: "127.16.94.133" port: 42559 } attrs { promote: true } } }
W20240503 14:13:43.566984 20774 consensus_peers.cc:485] T 0eb7552695304a1ea6f8e02f189f7295 P d6bcbc9fecfb44fa99a81917ea2d559a -> Peer f8c99fddf5974401b64aa8601605b2b2 (127.16.94.133:42559): Couldn't send request to peer f8c99fddf5974401b64aa8601605b2b2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0eb7552695304a1ea6f8e02f189f7295. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:43.861253 21030 ts_tablet_manager.cc:902] T ad711d78ee524d979867b65906abe456 P c4d0f91a34804de99e3a2a255cd06aed: Initiating tablet copy from peer a5d66df07715448fab64928f5bc8299a (127.16.94.130:39295)
I20240503 14:13:43.862944 21030 tablet_copy_client.cc:308] T ad711d78ee524d979867b65906abe456 P c4d0f91a34804de99e3a2a255cd06aed: tablet copy: Beginning tablet copy session from remote peer at address 127.16.94.130:39295
I20240503 14:13:43.863586 20271 tablet_copy_service.cc:140] P a5d66df07715448fab64928f5bc8299a: Received BeginTabletCopySession request for tablet ad711d78ee524d979867b65906abe456 from peer c4d0f91a34804de99e3a2a255cd06aed ({username='slave'} at 127.0.0.1:38712)
I20240503 14:13:43.863756 20271 tablet_copy_service.cc:161] P a5d66df07715448fab64928f5bc8299a: Beginning new tablet copy session on tablet ad711d78ee524d979867b65906abe456 from peer c4d0f91a34804de99e3a2a255cd06aed at {username='slave'} at 127.0.0.1:38712: session id = c4d0f91a34804de99e3a2a255cd06aed-ad711d78ee524d979867b65906abe456
I20240503 14:13:43.866305 20271 tablet_copy_source_session.cc:205] T ad711d78ee524d979867b65906abe456 P a5d66df07715448fab64928f5bc8299a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:43.867471 21030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad711d78ee524d979867b65906abe456. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:43.872880 21030 tablet_copy_client.cc:783] T ad711d78ee524d979867b65906abe456 P c4d0f91a34804de99e3a2a255cd06aed: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:43.873142 21030 tablet_copy_client.cc:647] T ad711d78ee524d979867b65906abe456 P c4d0f91a34804de99e3a2a255cd06aed: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:43.874580 21034 tablet_copy_client.cc:1002] T ad711d78ee524d979867b65906abe456 P c4d0f91a34804de99e3a2a255cd06aed: tablet copy: Injecting latency into file download: 2000
I20240503 14:13:43.914927 21035 ts_tablet_manager.cc:902] T 44a4d2b820fc4d0da06c659fe98fe57c P d6bcbc9fecfb44fa99a81917ea2d559a: Initiating tablet copy from peer a5d66df07715448fab64928f5bc8299a (127.16.94.130:39295)
F20240503 14:13:43.915263 21035 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (init.queue_time_us, init.queue_time_us) (init.run_cpu_time_us, init.run_cpu_time_us) (init.run_wall_time_us, init.run_wall_time_us) (leader-initialization.queue_time_us, leader-initialization.queue_time_us) (leader-initialization.run_cpu_time_us, leader-initialization.run_cpu_time_us) (leader-initialization.run_wall_time_us, leader-initialization.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-059a1f25723d4ba0be4cbb7185517bed.queue_time_us, tablet-download-pool-059a1f25723d4ba0be4cbb7185517bed.queue_time_us) (tablet-download-pool-059a1f25723d4ba0be4cbb7185517bed.run_cpu_time_us, tablet-download-pool-059a1f25723d4ba0be4cbb7185517bed.run_cpu_time_us) (tablet-download-pool-059a1f25723d4ba0be4cbb7185517bed.run_wall_time_us, tablet-download-pool-059a1f25723d4ba0be4cbb7185517bed.run_wall_time_us) (tablet-download-pool-0eb7552695304a1ea6f8e02f189f7295.queue_time_us, tablet-download-pool-0eb7552695304a1ea6f8e02f189f7295.queue_time_us) (tablet-download-pool-0eb7552695304a1ea6f8e02f189f7295.run_cpu_time_us, tablet-download-pool-0eb7552695304a1ea6f8e02f189f7295.run_cpu_time_us) (tablet-download-pool-0eb7552695304a1ea6f8e02f189f7295.run_wall_time_us, tablet-download-pool-0eb7552695304a1ea6f8e02f189f7295.run_wall_time_us) (tablet-download-pool-121af11b035d49dd8a2f12cc8eb7c583.queue_time_us, tablet-download-pool-121af11b035d49dd8a2f12cc8eb7c583.queue_time_us) (tablet-download-pool-121af11b035d49dd8a2f12cc8eb7c583.run_cpu_time_us, tablet-download-pool-121af11b035d49dd8a2f12cc8eb7c583.run_cpu_time_us) (tablet-download-pool-121af11b035d49dd8a2f12cc8eb7c583.run_wall_time_us, tablet-download-pool-121af11b035d49dd8a2f12cc8eb7c583.run_wall_time_us) (tablet-download-pool-220ef8bab4694ace9bdf0fce06226efa.queue_time_us, tablet-download-pool-220ef8bab4694ace9bdf0fce06226efa.queue_time_us) (tablet-download-pool-220ef8bab4694ace9bdf0fce06226efa.run_cpu_time_us, tablet-download-pool-220ef8bab4694ace9bdf0fce06226efa.run_cpu_time_us) (tablet-download-pool-220ef8bab4694ace9bdf0fce06226efa.run_wall_time_us, tablet-download-pool-220ef8bab4694ace9bdf0fce06226efa.run_wall_time_us) (tablet-download-pool-29f4ee03917646f6a5374217bed7a1be.queue_time_us, tablet-download-pool-29f4ee03917646f6a5374217bed7a1be.queue_time_us) (tablet-download-pool-29f4ee03917646f6a5374217bed7a1be.run_cpu_time_us, tablet-download-pool-29f4ee03917646f6a5374217bed7a1be.run_cpu_time_us) (tablet-download-pool-29f4ee03917646f6a5374217bed7a1be.run_wall_time_us, tablet-download-pool-29f4ee03917646f6a5374217bed7a1be.run_wall_time_us) (tablet-download-pool-3054a11a8da4470ea5241b92d164d6a8.queue_time_us, tablet-download-pool-3054a11a8da4470ea5241b92d164d6a8.queue_time_us) (tablet-download-pool-3054a11a8da4470ea5241b92d164d6a8.run_cpu_time_us, tablet-download-pool-3054a11a8da4470ea5241b92d164d6a8.run_cpu_time_us) (tablet-download-pool-3054a11a8da4470ea5241b92d164d6a8.run_wall_time_us, tablet-download-pool-3054a11a8da4470ea5241b92d164d6a8.run_wall_time_us) (tablet-download-pool-340236565cbb4796ad23ed5d13c9f14e.queue_time_us, tablet-download-pool-340236565cbb4796ad23ed5d13c9f14e.queue_time_us) (tablet-download-pool-340236565cbb4796ad23ed5d13c9f14e.run_cpu_time_us, tablet-download-pool-340236565cbb4796ad23ed5d13c9f14e.run_cpu_time_us) (tablet-download-pool-340236565cbb4796ad23ed5d13c9f14e.run_wall_time_us, tablet-download-pool-340236565cbb4796ad23ed5d13c9f14e.run_wall_time_us) (tablet-download-pool-44a4d2b820fc4d0da06c659fe98fe57c.queue_time_us, tablet-download-pool-44a4d2b820fc4d0da06c659fe98fe57c.queue_time_us) (tablet-download-pool-904450ab5e8b4250b7af18b98d2b0985.queue_time_us, tablet-download-pool-904450ab5e8b4250b7af18b98d2b0985.queue_time_us) (tablet-download-pool-904450ab5e8b4250b7af18b98d2b0985.run_cpu_time_us, tablet-download-pool-904450ab5e8b4250b7af18b98d2b0985.run_cpu_time_us) (tablet-download-pool-904450ab5e8b4250b7af18b98d2b0985.run_wall_time_us, tablet-download-pool-904450ab5e8b4250b7af18b98d2b0985.run_wall_time_us) (tablet-download-pool-ad711d78ee524d979867b65906abe456.queue_time_us, tablet-download-pool-ad711d78ee524d979867b65906abe456.queue_time_us) (tablet-download-pool-ad711d78ee524d979867b65906abe456.run_cpu_time_us, tablet-download-pool-ad711d78ee524d979867b65906abe456.run_cpu_time_us) (tablet-download-pool-ad711d78ee524d979867b65906abe456.run_wall_time_us, tablet-download-pool-ad711d78ee524d979867b65906abe456.run_wall_time_us) (tablet-download-pool-c89a88f56fb54d10be8eb56f67b3a8ce.queue_time_us, tablet-download-pool-c89a88f56fb54d10be8eb56f67b3a8ce.queue_time_us) (tablet-download-pool-c89a88f56fb54d10be8eb56f67b3a8ce.run_cpu_time_us, tablet-download-pool-c89a88f56fb54d10be8eb56f67b3a8ce.run_cpu_time_us) (tablet-download-pool-c89a88f56fb54d10be8eb56f67b3a8ce.run_wall_time_us, tablet-download-pool-c89a88f56fb54d10be8eb56f67b3a8ce.run_wall_time_us) (tablet-download-pool-ce7190aa241648008185b18f4a5b881f.queue_time_us, tablet-download-pool-ce7190aa241648008185b18f4a5b881f.queue_time_us) (tablet-download-pool-ce7190aa241648008185b18f4a5b881f.run_cpu_time_us, tablet-download-pool-ce7190aa241648008185b18f4a5b881f.run_cpu_time_us) (tablet-download-pool-ce7190aa241648008185b18f4a5b881f.run_wall_time_us, tablet-download-pool-ce7190aa241648008185b18f4a5b881f.run_wall_time_us) (tablet-download-pool-d4743d5a73c54086b277f4c4b1a248fd.queue_time_us, tablet-download-pool-d4743d5a73c54086b277f4c4b1a248fd.queue_time_us) (tablet-download-pool-d4743d5a73c54086b277f4c4b1a248fd.run_cpu_time_us, tablet-download-pool-d4743d5a73c54086b277f4c4b1a248fd.run_cpu_time_us) (tablet-download-pool-d4743d5a73c54086b277f4c4b1a248fd.run_wall_time_us, tablet-download-pool-d4743d5a73c54086b277f4c4b1a248fd.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
*** Aborted at 1714745623 (unix time) try "date -d @1714745623" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e80000417a) received by PID 16762 (TID 0x7f1c044d0700) from PID 16762; stack trace: ***
@ 0x7f1cb1f2e980 (unknown) at ??:0
@ 0x7f1ca727bfb7 gsignal at ??:0
@ 0x7f1ca727d921 abort at ??:0
@ 0x7f1ca9d30dcd google::LogMessage::Fail() at ??:0
@ 0x7f1ca9d34b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f1ca9d307cc google::LogMessage::Flush() at ??:0
@ 0x7f1ca9d31f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f1cab3ad742 kudu::TraceMetrics::InternName() at ??:0
@ 0x7f1cab379f8b kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f1cab373cd6 kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f1cba8408de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f1cba858daf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7f1cba983480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7f1cba9bba8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f1cba99e981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
I20240503 14:13:44.000144 21051 ts_tablet_manager.cc:902] T 7a82ae1f4c66455caea352a304af2b51 P f8c99fddf5974401b64aa8601605b2b2: Initiating tablet copy from peer 64ad8799c8df4b7589869baffb93b68d (127.16.94.129:37429)
@ 0x7f1cba99e70e std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f1cbf84098d std::function<>::operator()() at ??:0
@ 0x7f1cab37db75 kudu::ThreadPool::DispatchThread() at ??:0
I20240503 14:13:44.016418 21055 ts_tablet_manager.cc:902] T 0eb7552695304a1ea6f8e02f189f7295 P f8c99fddf5974401b64aa8601605b2b2: Initiating tablet copy from peer d6bcbc9fecfb44fa99a81917ea2d559a (127.16.94.134:34983)
@ 0x7f1cab38119c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
I20240503 14:13:44.018985 21057 ts_tablet_manager.cc:902] T c45ceab3a36d4225aed74654ed530c9d P c4d0f91a34804de99e3a2a255cd06aed: Initiating tablet copy from peer 64ad8799c8df4b7589869baffb93b68d (127.16.94.129:37429)
@ 0x7f1cab381019 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f1cbf84098d std::function<>::operator()() at ??:0
@ 0x7f1cab35c01e kudu::Thread::SuperviseThread() at ??:0
@ 0x7f1cb1f236db start_thread at ??:0
@ 0x7f1ca735e71f clone at ??:0