[==========] Running 14 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 14 tests from AutoRebalancerTest
[ RUN ] AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240502 08:02:13.787806 6963 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.6.204.254:35441,127.6.204.253:42053,127.6.204.252:45599
I20240502 08:02:13.789372 6963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:02:13.790380 6963 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:02:13.807507 6970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:02:13.807667 6972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:02:13.809216 6963 server_base.cc:1034] running on GCE node
W20240502 08:02:13.807494 6969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:02:14.931097 6963 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:02:14.931339 6963 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:02:14.931545 6963 hybrid_clock.cc:648] HybridClock initialized: now 1714636934931522 us; error 0 us; skew 500 ppm
I20240502 08:02:14.932386 6963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:02:14.937753 6963 webserver.cc:460] Webserver started at http://127.6.204.254:34841/ using document root <none> and password file <none>
I20240502 08:02:14.938592 6963 fs_manager.cc:359] Metadata directory not provided
I20240502 08:02:14.938812 6963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:02:14.939229 6963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:02:14.945549 6963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714636933625983-6963-0/minicluster-data/master-0-root/instance:
uuid: "5a6bf0a0ba2f44bb879ba72ea9378a12"
format_stamp: "Formatted at 2024-05-02 08:02:14 on dist-test-slave-32xs"
I20240502 08:02:14.953222 6963 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20240502 08:02:14.958338 6978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:02:14.959439 6963 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20240502 08:02:14.959766 6963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714636933625983-6963-0/minicluster-data/master-0-root
uuid: "5a6bf0a0ba2f44bb879ba72ea9378a12"
format_stamp: "Formatted at 2024-05-02 08:02:14 on dist-test-slave-32xs"
I20240502 08:02:14.960110 6963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714636933625983-6963-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714636933625983-6963-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714636933625983-6963-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:02:15.014065 6963 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:02:15.015671 6963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:02:15.016144 6963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:02:15.087774 6963 rpc_server.cc:307] RPC server started. Bound to: 127.6.204.254:35441
I20240502 08:02:15.087867 7029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.204.254:35441 every 8 connection(s)
I20240502 08:02:15.093791 6963 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 08:02:15.094713 7030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:02:15.101188 7032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:02:15.102682 7033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:02:15.108970 7035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:02:15.109036 6963 server_base.cc:1034] running on GCE node
I20240502 08:02:15.109968 6963 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:02:15.110282 6963 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:02:15.110507 6963 hybrid_clock.cc:648] HybridClock initialized: now 1714636935110489 us; error 0 us; skew 500 ppm
I20240502 08:02:15.111259 6963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:02:15.115514 6963 webserver.cc:460] Webserver started at http://127.6.204.253:41751/ using document root <none> and password file <none>
I20240502 08:02:15.116211 6963 fs_manager.cc:359] Metadata directory not provided
I20240502 08:02:15.116472 6963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:02:15.116829 6963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:02:15.118291 6963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714636933625983-6963-0/minicluster-data/master-1-root/instance:
uuid: "40039c085f394e6c966aeee46f9425fb"
format_stamp: "Formatted at 2024-05-02 08:02:15 on dist-test-slave-32xs"
I20240502 08:02:15.122291 7030 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.204.254" port: 35441 } has no permanent_uuid. Determining permanent_uuid...
I20240502 08:02:15.125051 6963 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20240502 08:02:15.130892 7041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:02:15.131875 6963 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20240502 08:02:15.132206 6963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714636933625983-6963-0/minicluster-data/master-1-root
uuid: "40039c085f394e6c966aeee46f9425fb"
format_stamp: "Formatted at 2024-05-02 08:02:15 on dist-test-slave-32xs"
I20240502 08:02:15.132694 6963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714636933625983-6963-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714636933625983-6963-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714636933625983-6963-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:02:15.151630 7030 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.204.253" port: 42053 } has no permanent_uuid. Determining permanent_uuid...
W20240502 08:02:15.154021 6979 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.204.253:42053: connect: Connection refused (error 111)
W20240502 08:02:15.158026 7030 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.6.204.253:42053: Network error: Client connection negotiation failed: client connection to 127.6.204.253:42053: connect: Connection refused (error 111)
I20240502 08:02:15.160773 6963 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:02:15.162106 6963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:02:15.207592 7030 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.204.253" port: 42053 } attempt: 1
W20240502 08:02:15.212301 7030 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.6.204.253:42053: Network error: Client connection negotiation failed: client connection to 127.6.204.253:42053: connect: Connection refused (error 111)
I20240502 08:02:15.236008 6963 rpc_server.cc:307] RPC server started. Bound to: 127.6.204.253:42053
I20240502 08:02:15.236187 7094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.204.253:42053 every 8 connection(s)
I20240502 08:02:15.241194 7095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:02:15.242156 6963 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 08:02:15.247968 7095 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.204.254" port: 35441 } has no permanent_uuid. Determining permanent_uuid...
W20240502 08:02:15.248512 7097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:02:15.254489 7098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:02:15.258406 6963 server_base.cc:1034] running on GCE node
W20240502 08:02:15.258540 7100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:02:15.259500 6963 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:02:15.259773 6963 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:02:15.259982 6963 hybrid_clock.cc:648] HybridClock initialized: now 1714636935259965 us; error 0 us; skew 500 ppm
I20240502 08:02:15.260605 6963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:02:15.264662 6963 webserver.cc:460] Webserver started at http://127.6.204.252:41031/ using document root <none> and password file <none>
I20240502 08:02:15.265261 6963 fs_manager.cc:359] Metadata directory not provided
I20240502 08:02:15.265470 6963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:02:15.265739 6963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:02:15.265977 7095 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.204.253" port: 42053 } has no permanent_uuid. Determining permanent_uuid...
I20240502 08:02:15.267043 6963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714636933625983-6963-0/minicluster-data/master-2-root/instance:
uuid: "62726479f252481b8f57e1dcba2d58b0"
format_stamp: "Formatted at 2024-05-02 08:02:15 on dist-test-slave-32xs"
I20240502 08:02:15.274016 6963 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20240502 08:02:15.277500 7095 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.204.252" port: 45599 } has no permanent_uuid. Determining permanent_uuid...
I20240502 08:02:15.279255 7107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:02:15.280117 6963 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20240502 08:02:15.280405 6963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714636933625983-6963-0/minicluster-data/master-2-root
uuid: "62726479f252481b8f57e1dcba2d58b0"
format_stamp: "Formatted at 2024-05-02 08:02:15 on dist-test-slave-32xs"
I20240502 08:02:15.280705 6963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714636933625983-6963-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714636933625983-6963-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714636933625983-6963-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:02:15.280896 7030 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.204.253" port: 42053 } attempt: 2
W20240502 08:02:15.283351 7095 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.6.204.252:45599: Network error: Client connection negotiation failed: client connection to 127.6.204.252:45599: connect: Connection refused (error 111)
I20240502 08:02:15.291568 7030 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.204.252" port: 45599 } has no permanent_uuid. Determining permanent_uuid...
W20240502 08:02:15.295539 7030 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.6.204.252:45599: Network error: Client connection negotiation failed: client connection to 127.6.204.252:45599: connect: Connection refused (error 111)
I20240502 08:02:15.305300 6963 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:02:15.306660 6963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:02:15.326848 7095 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.204.252" port: 45599 } attempt: 1
I20240502 08:02:15.326995 7030 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.204.252" port: 45599 } attempt: 1
W20240502 08:02:15.332286 7030 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.6.204.252:45599: Network error: Client connection negotiation failed: client connection to 127.6.204.252:45599: connect: Connection refused (error 111)
W20240502 08:02:15.332499 7095 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.6.204.252:45599: Network error: Client connection negotiation failed: client connection to 127.6.204.252:45599: connect: Connection refused (error 111)
I20240502 08:02:15.375974 6963 rpc_server.cc:307] RPC server started. Bound to: 127.6.204.252:45599
I20240502 08:02:15.376080 7159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.204.252:45599 every 8 connection(s)
I20240502 08:02:15.380323 6963 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 08:02:15.381484 7160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:02:15.386816 7160 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.204.254" port: 35441 } has no permanent_uuid. Determining permanent_uuid...
I20240502 08:02:15.397964 7160 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.204.253" port: 42053 } has no permanent_uuid. Determining permanent_uuid...
I20240502 08:02:15.399983 7095 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.204.252" port: 45599 } attempt: 2
I20240502 08:02:15.407853 7030 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.204.252" port: 45599 } attempt: 2
I20240502 08:02:15.409930 7160 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.204.252" port: 45599 } has no permanent_uuid. Determining permanent_uuid...
I20240502 08:02:15.423290 7095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40039c085f394e6c966aeee46f9425fb: Bootstrap starting.
I20240502 08:02:15.428910 7095 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 40039c085f394e6c966aeee46f9425fb: Neither blocks nor log segments found. Creating new log.
I20240502 08:02:15.429813 7160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62726479f252481b8f57e1dcba2d58b0: Bootstrap starting.
I20240502 08:02:15.430379 7030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a6bf0a0ba2f44bb879ba72ea9378a12: Bootstrap starting.
I20240502 08:02:15.431468 7095 log.cc:824] T 00000000000000000000000000000000 P 40039c085f394e6c966aeee46f9425fb: Log is configured to *not* fsync() on all Append() calls
I20240502 08:02:15.436205 7160 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62726479f252481b8f57e1dcba2d58b0: Neither blocks nor log segments found. Creating new log.
I20240502 08:02:15.436205 7030 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a6bf0a0ba2f44bb879ba72ea9378a12: Neither blocks nor log segments found. Creating new log.
I20240502 08:02:15.436877 7095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40039c085f394e6c966aeee46f9425fb: No bootstrap required, opened a new log
I20240502 08:02:15.443410 7030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a6bf0a0ba2f44bb879ba72ea9378a12: No bootstrap required, opened a new log
I20240502 08:02:15.443923 7160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62726479f252481b8f57e1dcba2d58b0: No bootstrap required, opened a new log
I20240502 08:02:15.458809 7095 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40039c085f394e6c966aeee46f9425fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a6bf0a0ba2f44bb879ba72ea9378a12" member_type: VOTER last_known_addr { host: "127.6.204.254" port: 35441 } } peers { permanent_uuid: "40039c085f394e6c966aeee46f9425fb" member_type: VOTER last_known_addr { host: "127.6.204.253" port: 42053 } } peers { permanent_uuid: "62726479f252481b8f57e1dcba2d58b0" member_type: VOTER last_known_addr { host: "127.6.204.252" port: 45599 } }
I20240502 08:02:15.458722 7030 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a6bf0a0ba2f44bb879ba72ea9378a12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a6bf0a0ba2f44bb879ba72ea9378a12" member_type: VOTER last_known_addr { host: "127.6.204.254" port: 35441 } } peers { permanent_uuid: "40039c085f394e6c966aeee46f9425fb" member_type: VOTER last_known_addr { host: "127.6.204.253" port: 42053 } } peers { permanent_uuid: "62726479f252481b8f57e1dcba2d58b0" member_type: VOTER last_known_addr { host: "127.6.204.252" port: 45599 } }
I20240502 08:02:15.458722 7160 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62726479f252481b8f57e1dcba2d58b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a6bf0a0ba2f44bb879ba72ea9378a12" member_type: VOTER last_known_addr { host: "127.6.204.254" port: 35441 } } peers { permanent_uuid: "40039c085f394e6c966aeee46f9425fb" member_type: VOTER last_known_addr { host: "127.6.204.253" port: 42053 } } peers { permanent_uuid: "62726479f252481b8f57e1dcba2d58b0" member_type: VOTER last_known_addr { host: "127.6.204.252" port: 45599 } }
I20240502 08:02:15.459704 7095 raft_consensus.cc:374] T 00000000000000000000000000000000 P 40039c085f394e6c966aeee46f9425fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:02:15.459810 7030 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a6bf0a0ba2f44bb879ba72ea9378a12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:02:15.459834 7160 raft_consensus.cc:374] T 00000000000000000000000000000000 P 62726479f252481b8f57e1dcba2d58b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:02:15.460062 7095 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40039c085f394e6c966aeee46f9425fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40039c085f394e6c966aeee46f9425fb, State: Initialized, Role: FOLLOWER
I20240502 08:02:15.460253 7160 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62726479f252481b8f57e1dcba2d58b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62726479f252481b8f57e1dcba2d58b0, State: Initialized, Role: FOLLOWER
I20240502 08:02:15.460215 7030 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a6bf0a0ba2f44bb879ba72ea9378a12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a6bf0a0ba2f44bb879ba72ea9378a12, State: Initialized, Role: FOLLOWER
I20240502 08:02:15.461192 7095 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40039c085f394e6c966aeee46f9425fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a6bf0a0ba2f44bb879ba72ea9378a12" member_type: VOTER last_known_addr { host: "127.6.204.254" port: 35441 } } peers { permanent_uuid: "40039c085f394e6c966aeee46f9425fb" member_type: VOTER last_known_addr { host: "127.6.204.253" port: 42053 } } peers { permanent_uuid: "62726479f252481b8f57e1dcba2d58b0" member_type: VOTER last_known_addr { host: "127.6.204.252" port: 45599 } }
I20240502 08:02:15.461228 7160 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62726479f252481b8f57e1dcba2d58b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a6bf0a0ba2f44bb879ba72ea9378a12" member_type: VOTER last_known_addr { host: "127.6.204.254" port: 35441 } } peers { permanent_uuid: "40039c085f394e6c966aeee46f9425fb" member_type: VOTER last_known_addr { host: "127.6.204.253" port: 42053 } } peers { permanent_uuid: "62726479f252481b8f57e1dcba2d58b0" member_type: VOTER last_known_addr { host: "127.6.204.252" port: 45599 } }
I20240502 08:02:15.461228 7030 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a6bf0a0ba2f44bb879ba72ea9378a12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a6bf0a0ba2f44bb879ba72ea9378a12" member_type: VOTER last_known_addr { host: "127.6.204.254" port: 35441 } } peers { permanent_uuid: "40039c085f394e6c966aeee46f9425fb" member_type: VOTER last_known_addr { host: "127.6.204.253" port: 42053 } } peers { permanent_uuid: "62726479f252481b8f57e1dcba2d58b0" member_type: VOTER last_known_addr { host: "127.6.204.252" port: 45599 } }
I20240502 08:02:15.463423 7169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40039c085f394e6c966aeee46f9425fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a6bf0a0ba2f44bb879ba72ea9378a12" member_type: VOTER last_known_addr { host: "127.6.204.254" port: 35441 } } peers { permanent_uuid: "40039c085f394e6c966aeee46f9425fb" member_type: VOTER last_known_addr { host: "127.6.204.253" port: 42053 } } peers { permanent_uuid: "62726479f252481b8f57e1dcba2d58b0" member_type: VOTER last_known_addr { host: "127.6.204.252" port: 45599 } } }
I20240502 08:02:15.464447 7169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40039c085f394e6c966aeee46f9425fb [sys.catalog]: This master's current role is: FOLLOWER
I20240502 08:02:15.465413 7095 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40039c085f394e6c966aeee46f9425fb [sys.catalog]: configured and running, proceeding with master startup.
I20240502 08:02:15.465663 7171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a6bf0a0ba2f44bb879ba72ea9378a12 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a6bf0a0ba2f44bb879ba72ea9378a12" member_type: VOTER last_known_addr { host: "127.6.204.254" port: 35441 } } peers { permanent_uuid: "40039c085f394e6c966aeee46f9425fb" member_type: VOTER last_known_addr { host: "127.6.204.253" port: 42053 } } peers { permanent_uuid: "62726479f252481b8f57e1dcba2d58b0" member_type: VOTER last_known_addr { host: "127.6.204.252" port: 45599 } } }
I20240502 08:02:15.466552 7171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a6bf0a0ba2f44bb879ba72ea9378a12 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 08:02:15.467437 7030 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a6bf0a0ba2f44bb879ba72ea9378a12 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 08:02:15.468081 7170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62726479f252481b8f57e1dcba2d58b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a6bf0a0ba2f44bb879ba72ea9378a12" member_type: VOTER last_known_addr { host: "127.6.204.254" port: 35441 } } peers { permanent_uuid: "40039c085f394e6c966aeee46f9425fb" member_type: VOTER last_known_addr { host: "127.6.204.253" port: 42053 } } peers { permanent_uuid: "62726479f252481b8f57e1dcba2d58b0" member_type: VOTER last_known_addr { host: "127.6.204.252" port: 45599 } } }
I20240502 08:02:15.468997 7170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62726479f252481b8f57e1dcba2d58b0 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 08:02:15.469506 7160 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62726479f252481b8f57e1dcba2d58b0 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 08:02:15.509457 6963 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
I20240502 08:02:15.510712 6963 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
W20240502 08:02:15.517807 7200 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 40039c085f394e6c966aeee46f9425fb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240502 08:02:15.517930 7201 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5a6bf0a0ba2f44bb879ba72ea9378a12: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240502 08:02:15.518260 7200 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20240502 08:02:15.518430 7201 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20240502 08:02:15.521719 6984 tablet.cc:2369] T 00000000000000000000000000000000 P 5a6bf0a0ba2f44bb879ba72ea9378a12: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 08:02:15.523442 7169 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40039c085f394e6c966aeee46f9425fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:02:15.523998 6963 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 08:02:15.523978 7169 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40039c085f394e6c966aeee46f9425fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a6bf0a0ba2f44bb879ba72ea9378a12" member_type: VOTER last_known_addr { host: "127.6.204.254" port: 35441 } } peers { permanent_uuid: "40039c085f394e6c966aeee46f9425fb" member_type: VOTER last_known_addr { host: "127.6.204.253" port: 42053 } } peers { permanent_uuid: "62726479f252481b8f57e1dcba2d58b0" member_type: VOTER last_known_addr { host: "127.6.204.252" port: 45599 } }
W20240502 08:02:15.527251 7205 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 62726479f252481b8f57e1dcba2d58b0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240502 08:02:15.527570 7205 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240502 08:02:15.528033 7005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "40039c085f394e6c966aeee46f9425fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a6bf0a0ba2f44bb879ba72ea9378a12" is_pre_election: true
I20240502 08:02:15.528676 7169 leader_election.cc:288] T 00000000000000000000000000000000 P 40039c085f394e6c966aeee46f9425fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a6bf0a0ba2f44bb879ba72ea9378a12 (127.6.204.254:35441), 62726479f252481b8f57e1dcba2d58b0 (127.6.204.252:45599)
I20240502 08:02:15.528910 7005 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5a6bf0a0ba2f44bb879ba72ea9378a12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40039c085f394e6c966aeee46f9425fb in term 0.
I20240502 08:02:15.529691 7135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "40039c085f394e6c966aeee46f9425fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62726479f252481b8f57e1dcba2d58b0" is_pre_election: true
I20240502 08:02:15.530442 7135 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 62726479f252481b8f57e1dcba2d58b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40039c085f394e6c966aeee46f9425fb in term 0.
I20240502 08:02:15.530403 7044 leader_election.cc:302] T 00000000000000000000000000000000 P 40039c085f394e6c966aeee46f9425fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40039c085f394e6c966aeee46f9425fb, 5a6bf0a0ba2f44bb879ba72ea9378a12; no voters:
I20240502 08:02:15.531694 7169 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40039c085f394e6c966aeee46f9425fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:02:15.532191 7169 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40039c085f394e6c966aeee46f9425fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:02:15.532624 7169 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40039c085f394e6c966aeee46f9425fb [term 0 FOLLOWER]: Advancing to term 1
W20240502 08:02:15.537055 7206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:02:15.537770 7207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:02:15.541410 7169 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40039c085f394e6c966aeee46f9425fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a6bf0a0ba2f44bb879ba72ea9378a12" member_type: VOTER last_known_addr { host: "127.6.204.254" port: 35441 } } peers { permanent_uuid: "40039c085f394e6c966aeee46f9425fb" member_type: VOTER last_known_addr { host: "127.6.204.253" port: 42053 } } peers { permanent_uuid: "62726479f252481b8f57e1dcba2d58b0" member_type: VOTER last_known_addr { host: "127.6.204.252" port: 45599 } }
I20240502 08:02:15.542290 6963 server_base.cc:1034] running on GCE node
W20240502 08:02:15.543051 7209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:02:15.543406 7169 leader_election.cc:288] T 00000000000000000000000000000000 P 40039c085f394e6c966aeee46f9425fb [CANDIDATE]: Term 1 election: Requested vote from peers 5a6bf0a0ba2f44bb879ba72ea9378a12 (127.6.204.254:35441), 62726479f252481b8f57e1dcba2d58b0 (127.6.204.252:45599)
I20240502 08:02:15.544188 6963 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:02:15.544519 6963 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:02:15.544832 6963 hybrid_clock.cc:648] HybridClock initialized: now 1714636935544811 us; error 0 us; skew 500 ppm
I20240502 08:02:15.544510 7135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "40039c085f394e6c966aeee46f9425fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62726479f252481b8f57e1dcba2d58b0"
I20240502 08:02:15.544510 7005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "40039c085f394e6c966aeee46f9425fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a6bf0a0ba2f44bb879ba72ea9378a12"
I20240502 08:02:15.545343 7135 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62726479f252481b8f57e1dcba2d58b0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:02:15.545380 7005 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a6bf0a0ba2f44bb879ba72ea9378a12 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:02:15.545651 6963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:02:15.551996 7005 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5a6bf0a0ba2f44bb879ba72ea9378a12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40039c085f394e6c966aeee46f9425fb in term 1.
I20240502 08:02:15.551997 7135 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 62726479f252481b8f57e1dcba2d58b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40039c085f394e6c966aeee46f9425fb in term 1.
I20240502 08:02:15.553517 7044 leader_election.cc:302] T 00000000000000000000000000000000 P 40039c085f394e6c966aeee46f9425fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40039c085f394e6c966aeee46f9425fb, 5a6bf0a0ba2f44bb879ba72ea9378a12; no voters:
I20240502 08:02:15.554160 6963 webserver.cc:460] Webserver started at http://127.6.204.193:43783/ using document root <none> and password file <none>
I20240502 08:02:15.554338 7169 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40039c085f394e6c966aeee46f9425fb [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:02:15.554819 6963 fs_manager.cc:359] Metadata directory not provided
I20240502 08:02:15.555064 6963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:02:15.555398 6963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:02:15.555444 7169 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40039c085f394e6c966aeee46f9425fb [term 1 LEADER]: Becoming Leader. State: Replica: 40039c085f394e6c966aeee46f9425fb, State: Running, Role: LEADER
I20240502 08:02:15.556377 7169 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40039c085f394e6c966aeee46f9425fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a6bf0a0ba2f44bb879ba72ea9378a12" member_type: VOTER last_known_addr { host: "127.6.204.254" port: 35441 } } peers { permanent_uuid: "40039c085f394e6c966aeee46f9425fb" member_type: VOTER last_known_addr { host: "127.6.204.253" port: 42053 } } peers { permanent_uuid: "62726479f252481b8f57e1dcba2d58b0" member_type: VOTER last_known_addr { host: "127.6.204.252" port: 45599 } }
I20240502 08:02:15.556911 6963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714636933625983-6963-0/minicluster-data/ts-0-root/instance:
uuid: "c56a85c19c0841ec9899441418fd5ed2"
format_stamp: "Formatted at 2024-05-02 08:02:15 on dist-test-slave-32xs"
I20240502 08:02:15.566329 6963 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.007s sys 0.005s
I20240502 08:02:15.566395 7213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40039c085f394e6c966aeee46f9425fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40039c085f394e6c966aeee46f9425fb. Latest consensus state: current_term: 1 leader_uuid: "40039c085f394e6c966aeee46f9425fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a6bf0a0ba2f44bb879ba72ea9378a12" member_type: VOTER last_known_addr { host: "127.6.204.254" port: 35441 } } peers { permanent_uuid: "40039c085f394e6c966aeee46f9425fb" member_type: VOTER last_known_addr { host: "127.6.204.253" port: 42053 } } peers { permanent_uuid: "62726479f252481b8f57e1dcba2d58b0" member_type: VOTER last_known_addr { host: "127.6.204.252" port: 45599 } } }
I20240502 08:02:15.567209 7213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40039c085f394e6c966aeee46f9425fb [sys.catalog]: This master's current role is: LEADER
I20240502 08:02:15.570019 7216 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 08:02:15.575171 7217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:02:15.576066 7216 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 08:02:15.576172 6963 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.005s sys 0.003s
I20240502 08:02:15.576540 6963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714636933625983-6963-0/minicluster-data/ts-0-root
uuid: "c56a85c19c0841ec9899441418fd5ed2"
format_stamp: "Formatted at 2024-05-02 08:02:15 on dist-test-slave-32xs"
I20240502 08:02:15.576913 6963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714636933625983-6963-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714636933625983-6963-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714636933625983-6963-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:02:15.601455 7135 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 62726479f252481b8f57e1dcba2d58b0 [term 1 FOLLOWER]: Refusing update from remote peer 40039c085f394e6c966aeee46f9425fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:02:15.602370 7005 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5a6bf0a0ba2f44bb879ba72ea9378a12 [term 1 FOLLOWER]: Refusing update from remote peer 40039c085f394e6c966aeee46f9425fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:02:15.603399 7213 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 40039c085f394e6c966aeee46f9425fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "62726479f252481b8f57e1dcba2d58b0" member_type: VOTER last_known_addr { host: "127.6.204.252" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:02:15.604126 7169 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 40039c085f394e6c966aeee46f9425fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a6bf0a0ba2f44bb879ba72ea9378a12" member_type: VOTER last_known_addr { host: "127.6.204.254" port: 35441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:02:15.623370 7171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a6bf0a0ba2f44bb879ba72ea9378a12 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40039c085f394e6c966aeee46f9425fb. Latest consensus state: current_term: 1 leader_uuid: "40039c085f394e6c966aeee46f9425fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a6bf0a0ba2f44bb879ba72ea9378a12" member_type: VOTER last_known_addr { host: "127.6.204.254" port: 35441 } } peers { permanent_uuid: "40039c085f394e6c966aeee46f9425fb" member_type: VOTER last_known_addr { host: "127.6.204.253" port: 42053 } } peers { permanent_uuid: "62726479f252481b8f57e1dcba2d58b0" member_type: VOTER last_known_addr { host: "127.6.204.252" port: 45599 } } }
I20240502 08:02:15.624151 7171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a6bf0a0ba2f44bb879ba72ea9378a12 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 08:02:15.628101 6963 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:02:15.630447 6963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:02:15.631093 7221 mvcc.cc:205] Tried to move back new op lower bound from 7023152888207261696 to 7023152888046473216. Current Snapshot: MvccSnapshot[applied={T|T < 7023152888207261696}]
I20240502 08:02:15.637513 7213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40039c085f394e6c966aeee46f9425fb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "40039c085f394e6c966aeee46f9425fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a6bf0a0ba2f44bb879ba72ea9378a12" member_type: VOTER last_known_addr { host: "127.6.204.254" port: 35441 } } peers { permanent_uuid: "40039c085f394e6c966aeee46f9425fb" member_type: VOTER last_known_addr { host: "127.6.204.253" port: 42053 } } peers { permanent_uuid: "62726479f252481b8f57e1dcba2d58b0" member_type: VOTER last_known_addr { host: "127.6.204.252" port: 45599 } } }
I20240502 08:02:15.638360 7213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40039c085f394e6c966aeee46f9425fb [sys.catalog]: This master's current role is: LEADER
I20240502 08:02:15.639568 7171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a6bf0a0ba2f44bb879ba72ea9378a12 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "40039c085f394e6c966aeee46f9425fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a6bf0a0ba2f44bb879ba72ea9378a12" member_type: VOTER last_known_addr { host: "127.6.204.254" port: 35441 } } peers { permanent_uuid: "40039c085f394e6c966aeee46f9425fb" member_type: VOTER last_known_addr { host: "127.6.204.253" port: 42053 } } peers { permanent_uuid: "62726479f252481b8f57e1dcba2d58b0" member_type: VOTER last_known_addr { host: "127.6.204.252" port: 45599 } } }
I20240502 08:02:15.640473 7171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a6bf0a0ba2f44bb879ba72ea9378a12 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 08:02:15.640780 7170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62726479f252481b8f57e1dcba2d58b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40039c085f394e6c966aeee46f9425fb. Latest consensus state: current_term: 1 leader_uuid: "40039c085f394e6c966aeee46f9425fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a6bf0a0ba2f44bb879ba72ea9378a12" member_type: VOTER last_known_addr { host: "127.6.204.254" port: 35441 } } peers { permanent_uuid: "40039c085f394e6c966aeee46f9425fb" member_type: VOTER last_known_addr { host: "127.6.204.253" port: 42053 } } peers { permanent_uuid: "62726479f252481b8f57e1dcba2d58b0" member_type: VOTER last_known_addr { host: "127.6.204.252" port: 45599 } } }
I20240502 08:02:15.641582 7170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62726479f252481b8f57e1dcba2d58b0 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 08:02:15.643039 7216 catalog_manager.cc:1347] Generated new cluster ID: db5bc5e227e04f189e634dcedaeb2ceb
I20240502 08:02:15.643360 7216 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 08:02:15.643968 6963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:02:15.645707 7169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40039c085f394e6c966aeee46f9425fb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "40039c085f394e6c966aeee46f9425fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a6bf0a0ba2f44bb879ba72ea9378a12" member_type: VOTER last_known_addr { host: "127.6.204.254" port: 35441 } } peers { permanent_uuid: "40039c085f394e6c966aeee46f9425fb" member_type: VOTER last_known_addr { host: "127.6.204.253" port: 42053 } } peers { permanent_uuid: "62726479f252481b8f57e1dcba2d58b0" member_type: VOTER last_known_addr { host: "127.6.204.252" port: 45599 } } }
I20240502 08:02:15.646484 7169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40039c085f394e6c966aeee46f9425fb [sys.catalog]: This master's current role is: LEADER
I20240502 08:02:15.649611 6963 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:02:15.649901 6963 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:02:15.650226 6963 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:02:15.650456 6963 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:02:15.657765 7170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62726479f252481b8f57e1dcba2d58b0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "40039c085f394e6c966aeee46f9425fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a6bf0a0ba2f44bb879ba72ea9378a12" member_type: VOTER last_known_addr { host: "127.6.204.254" port: 35441 } } peers { permanent_uuid: "40039c085f394e6c966aeee46f9425fb" member_type: VOTER last_known_addr { host: "127.6.204.253" port: 42053 } } peers { permanent_uuid: "62726479f252481b8f57e1dcba2d58b0" member_type: VOTER last_known_addr { host: "127.6.204.252" port: 45599 } } }
I20240502 08:02:15.658600 7170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62726479f252481b8f57e1dcba2d58b0 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 08:02:15.679694 7216 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 08:02:15.681922 7216 catalog_manager.cc:1504] Loading token signing keys...
I20240502 08:02:15.717839 7216 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 40039c085f394e6c966aeee46f9425fb: Generated new TSK 0
I20240502 08:02:15.719484 7216 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 08:02:15.786918 6963 rpc_server.cc:307] RPC server started. Bound to: 127.6.204.193:38389
I20240502 08:02:15.787189 7287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.204.193:38389 every 8 connection(s)
I20240502 08:02:15.820384 7288 heartbeater.cc:347] Connected to a master server at 127.6.204.252:45599
I20240502 08:02:15.821064 7288 heartbeater.cc:464] Registering TS with master...
I20240502 08:02:15.822629 7288 heartbeater.cc:511] Master 127.6.204.252:45599 requested a full tablet report, sending...
I20240502 08:02:15.827519 7125 ts_manager.cc:194] Registered new tserver with Master: c56a85c19c0841ec9899441418fd5ed2 (127.6.204.193:38389)
I20240502 08:02:15.827675 7289 heartbeater.cc:347] Connected to a master server at 127.6.204.253:42053
I20240502 08:02:15.828100 6963 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 08:02:15.828123 7289 heartbeater.cc:464] Registering TS with master...
I20240502 08:02:15.829250 7289 heartbeater.cc:511] Master 127.6.204.253:42053 requested a full tablet report, sending...
I20240502 08:02:15.831668 7060 ts_manager.cc:194] Registered new tserver with Master: c56a85c19c0841ec9899441418fd5ed2 (127.6.204.193:38389)
I20240502 08:02:15.834307 7291 heartbeater.cc:347] Connected to a master server at 127.6.204.254:35441
I20240502 08:02:15.834715 7291 heartbeater.cc:464] Registering TS with master...
I20240502 08:02:15.834658 7060 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45464
I20240502 08:02:15.835804 7291 heartbeater.cc:511] Master 127.6.204.254:35441 requested a full tablet report, sending...
W20240502 08:02:15.839118 7295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:02:15.839402 6995 ts_manager.cc:194] Registered new tserver with Master: c56a85c19c0841ec9899441418fd5ed2 (127.6.204.193:38389)
W20240502 08:02:15.844698 7296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:02:15.850572 6963 server_base.cc:1034] running on GCE node
W20240502 08:02:15.852563 7298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:02:15.853562 6963 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:02:15.853811 6963 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:02:15.853994 6963 hybrid_clock.cc:648] HybridClock initialized: now 1714636935853974 us; error 0 us; skew 500 ppm
I20240502 08:02:15.854523 6963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:02:15.857455 6963 webserver.cc:460] Webserver started at http://127.6.204.194:42907/ using document root <none> and password file <none>
I20240502 08:02:15.857934 6963 fs_manager.cc:359] Metadata directory not provided
I20240502 08:02:15.858160 6963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:02:15.858443 6963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:02:15.859539 6963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714636933625983-6963-0/minicluster-data/ts-1-root/instance:
uuid: "8f72ded747a044928707b7e44be40025"
format_stamp: "Formatted at 2024-05-02 08:02:15 on dist-test-slave-32xs"
I20240502 08:02:15.864748 6963 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240502 08:02:15.868458 7303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:02:15.869319 6963 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240502 08:02:15.869606 6963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714636933625983-6963-0/minicluster-data/ts-1-root
uuid: "8f72ded747a044928707b7e44be40025"
format_stamp: "Formatted at 2024-05-02 08:02:15 on dist-test-slave-32xs"
I20240502 08:02:15.869906 6963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714636933625983-6963-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714636933625983-6963-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714636933625983-6963-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:02:15.884433 6963 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:02:15.885651 6963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:02:15.888218 6963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:02:15.891621 6963 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:02:15.891808 6963 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:02:15.891992 6963 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:02:15.892127 6963 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:02:15.983402 6963 rpc_server.cc:307] RPC server started. Bound to: 127.6.204.194:43287
I20240502 08:02:15.983618 7365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.204.194:43287 every 8 connection(s)
I20240502 08:02:16.017246 7366 heartbeater.cc:347] Connected to a master server at 127.6.204.252:45599
I20240502 08:02:16.017673 7366 heartbeater.cc:464] Registering TS with master...
I20240502 08:02:16.018568 7366 heartbeater.cc:511] Master 127.6.204.252:45599 requested a full tablet report, sending...
I20240502 08:02:16.022102 6963 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 08:02:16.022262 7125 ts_manager.cc:194] Registered new tserver with Master: 8f72ded747a044928707b7e44be40025 (127.6.204.194:43287)
I20240502 08:02:16.025417 7369 heartbeater.cc:347] Connected to a master server at 127.6.204.254:35441
I20240502 08:02:16.025794 7369 heartbeater.cc:464] Registering TS with master...
I20240502 08:02:16.026723 7369 heartbeater.cc:511] Master 127.6.204.254:35441 requested a full tablet report, sending...
I20240502 08:02:16.027801 7367 heartbeater.cc:347] Connected to a master server at 127.6.204.253:42053
I20240502 08:02:16.028179 7367 heartbeater.cc:464] Registering TS with master...
I20240502 08:02:16.028872 7367 heartbeater.cc:511] Master 127.6.204.253:42053 requested a full tablet report, sending...
I20240502 08:02:16.031841 6995 ts_manager.cc:194] Registered new tserver with Master: 8f72ded747a044928707b7e44be40025 (127.6.204.194:43287)
W20240502 08:02:16.032171 7373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:02:16.034030 7060 ts_manager.cc:194] Registered new tserver with Master: 8f72ded747a044928707b7e44be40025 (127.6.204.194:43287)
W20240502 08:02:16.034996 7374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:02:16.037475 7060 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45468
W20240502 08:02:16.044503 7376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:02:16.044643 6963 server_base.cc:1034] running on GCE node
I20240502 08:02:16.045706 6963 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:02:16.045946 6963 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:02:16.046124 6963 hybrid_clock.cc:648] HybridClock initialized: now 1714636936046108 us; error 0 us; skew 500 ppm
I20240502 08:02:16.046653 6963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:02:16.049671 6963 webserver.cc:460] Webserver started at http://127.6.204.195:38579/ using document root <none> and password file <none>
I20240502 08:02:16.050163 6963 fs_manager.cc:359] Metadata directory not provided
I20240502 08:02:16.050369 6963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:02:16.050626 6963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:02:16.051666 6963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714636933625983-6963-0/minicluster-data/ts-2-root/instance:
uuid: "5d43c104f1ff4ef993228c4202aecc8e"
format_stamp: "Formatted at 2024-05-02 08:02:16 on dist-test-slave-32xs"
I20240502 08:02:16.056310 6963 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 08:02:16.060082 7381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:02:16.060951 6963 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240502 08:02:16.061326 6963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714636933625983-6963-0/minicluster-data/ts-2-root
uuid: "5d43c104f1ff4ef993228c4202aecc8e"
format_stamp: "Formatted at 2024-05-02 08:02:16 on dist-test-slave-32xs"
I20240502 08:02:16.061688 6963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714636933625983-6963-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714636933625983-6963-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714636933625983-6963-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:02:16.134204 6963 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:02:16.135535 6963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:02:16.137070 6963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:02:16.140656 6963 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:02:16.140851 6963 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:02:16.141135 6963 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:02:16.141369 6963 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:02:16.237519 6963 rpc_server.cc:307] RPC server started. Bound to: 127.6.204.195:40895
I20240502 08:02:16.237685 7443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.204.195:40895 every 8 connection(s)
I20240502 08:02:16.264686 7444 heartbeater.cc:347] Connected to a master server at 127.6.204.252:45599
I20240502 08:02:16.265108 7444 heartbeater.cc:464] Registering TS with master...
I20240502 08:02:16.266057 7444 heartbeater.cc:511] Master 127.6.204.252:45599 requested a full tablet report, sending...
I20240502 08:02:16.269280 7125 ts_manager.cc:194] Registered new tserver with Master: 5d43c104f1ff4ef993228c4202aecc8e (127.6.204.195:40895)
I20240502 08:02:16.274444 7445 heartbeater.cc:347] Connected to a master server at 127.6.204.253:42053
I20240502 08:02:16.274885 7445 heartbeater.cc:464] Registering TS with master...
I20240502 08:02:16.275868 7445 heartbeater.cc:511] Master 127.6.204.253:42053 requested a full tablet report, sending...
I20240502 08:02:16.276842 7447 heartbeater.cc:347] Connected to a master server at 127.6.204.254:35441
I20240502 08:02:16.277208 7447 heartbeater.cc:464] Registering TS with master...
I20240502 08:02:16.277892 7060 ts_manager.cc:194] Registered new tserver with Master: 5d43c104f1ff4ef993228c4202aecc8e (127.6.204.195:40895)
I20240502 08:02:16.278179 7447 heartbeater.cc:511] Master 127.6.204.254:35441 requested a full tablet report, sending...
I20240502 08:02:16.279516 7060 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45474
I20240502 08:02:16.280905 6995 ts_manager.cc:194] Registered new tserver with Master: 5d43c104f1ff4ef993228c4202aecc8e (127.6.204.195:40895)
I20240502 08:02:16.281265 6963 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.014344278s
I20240502 08:02:16.524192 7201 catalog_manager.cc:1259] Loaded cluster ID: db5bc5e227e04f189e634dcedaeb2ceb
I20240502 08:02:16.524533 7201 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5a6bf0a0ba2f44bb879ba72ea9378a12: loading cluster ID for follower catalog manager: success
I20240502 08:02:16.531327 7205 catalog_manager.cc:1259] Loaded cluster ID: db5bc5e227e04f189e634dcedaeb2ceb
I20240502 08:02:16.531579 7201 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5a6bf0a0ba2f44bb879ba72ea9378a12: acquiring CA information for follower catalog manager: success
I20240502 08:02:16.531620 7205 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 62726479f252481b8f57e1dcba2d58b0: loading cluster ID for follower catalog manager: success
I20240502 08:02:16.536610 7205 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 62726479f252481b8f57e1dcba2d58b0: acquiring CA information for follower catalog manager: success
I20240502 08:02:16.537307 7201 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5a6bf0a0ba2f44bb879ba72ea9378a12: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240502 08:02:16.540311 7205 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 62726479f252481b8f57e1dcba2d58b0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240502 08:02:16.840720 7289 heartbeater.cc:503] Master 127.6.204.253:42053 was elected leader, sending a full tablet report...
I20240502 08:02:17.040395 7367 heartbeater.cc:503] Master 127.6.204.253:42053 was elected leader, sending a full tablet report...
I20240502 08:02:17.281991 7445 heartbeater.cc:503] Master 127.6.204.253:42053 was elected leader, sending a full tablet report...
I20240502 08:02:17.506618 7197 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:02:17.585439 6963 test_util.cc:268] Using random seed: 93676282
I20240502 08:02:17.643270 7060 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45478:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:02:17.724785 7252 tablet_service.cc:1467] Processing CreateTablet for tablet 5630b3f83a6b48249dfd0269ef496ffd (DEFAULT_TABLE table=test-workload [id=2aae370a94394e5d94c2b6fab8c22898]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240502 08:02:17.726298 7331 tablet_service.cc:1467] Processing CreateTablet for tablet 14b4fa2a60874e81a585c6e84ca1f7f1 (DEFAULT_TABLE table=test-workload [id=2aae370a94394e5d94c2b6fab8c22898]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240502 08:02:17.726754 7252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5630b3f83a6b48249dfd0269ef496ffd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:02:17.724785 7253 tablet_service.cc:1467] Processing CreateTablet for tablet bafc82a647b24cda8cd0bd25c3c75f89 (DEFAULT_TABLE table=test-workload [id=2aae370a94394e5d94c2b6fab8c22898]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240502 08:02:17.727707 7331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14b4fa2a60874e81a585c6e84ca1f7f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:02:17.728089 7253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bafc82a647b24cda8cd0bd25c3c75f89. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:02:17.725025 7409 tablet_service.cc:1467] Processing CreateTablet for tablet 8a46172374b5455c9dd94e6c44701f66 (DEFAULT_TABLE table=test-workload [id=2aae370a94394e5d94c2b6fab8c22898]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240502 08:02:17.731529 7409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a46172374b5455c9dd94e6c44701f66. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:02:17.745541 7480 tablet_bootstrap.cc:492] T 14b4fa2a60874e81a585c6e84ca1f7f1 P 8f72ded747a044928707b7e44be40025: Bootstrap starting.
I20240502 08:02:17.752350 7479 tablet_bootstrap.cc:492] T bafc82a647b24cda8cd0bd25c3c75f89 P c56a85c19c0841ec9899441418fd5ed2: Bootstrap starting.
I20240502 08:02:17.754546 7480 tablet_bootstrap.cc:654] T 14b4fa2a60874e81a585c6e84ca1f7f1 P 8f72ded747a044928707b7e44be40025: Neither blocks nor log segments found. Creating new log.
I20240502 08:02:17.755476 7481 tablet_bootstrap.cc:492] T 8a46172374b5455c9dd94e6c44701f66 P 5d43c104f1ff4ef993228c4202aecc8e: Bootstrap starting.
I20240502 08:02:17.758078 7479 tablet_bootstrap.cc:654] T bafc82a647b24cda8cd0bd25c3c75f89 P c56a85c19c0841ec9899441418fd5ed2: Neither blocks nor log segments found. Creating new log.
I20240502 08:02:17.763406 7481 tablet_bootstrap.cc:654] T 8a46172374b5455c9dd94e6c44701f66 P 5d43c104f1ff4ef993228c4202aecc8e: Neither blocks nor log segments found. Creating new log.
I20240502 08:02:17.764695 7479 tablet_bootstrap.cc:492] T bafc82a647b24cda8cd0bd25c3c75f89 P c56a85c19c0841ec9899441418fd5ed2: No bootstrap required, opened a new log
I20240502 08:02:17.765197 7479 ts_tablet_manager.cc:1367] T bafc82a647b24cda8cd0bd25c3c75f89 P c56a85c19c0841ec9899441418fd5ed2: Time spent bootstrapping tablet: real 0.013s user 0.011s sys 0.000s
I20240502 08:02:17.766811 7480 tablet_bootstrap.cc:492] T 14b4fa2a60874e81a585c6e84ca1f7f1 P 8f72ded747a044928707b7e44be40025: No bootstrap required, opened a new log
I20240502 08:02:17.767267 7480 ts_tablet_manager.cc:1367] T 14b4fa2a60874e81a585c6e84ca1f7f1 P 8f72ded747a044928707b7e44be40025: Time spent bootstrapping tablet: real 0.022s user 0.010s sys 0.009s
I20240502 08:02:17.768095 7479 raft_consensus.cc:348] T bafc82a647b24cda8cd0bd25c3c75f89 P c56a85c19c0841ec9899441418fd5ed2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c56a85c19c0841ec9899441418fd5ed2" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38389 } }
I20240502 08:02:17.768703 7479 raft_consensus.cc:374] T bafc82a647b24cda8cd0bd25c3c75f89 P c56a85c19c0841ec9899441418fd5ed2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:02:17.769007 7479 raft_consensus.cc:729] T bafc82a647b24cda8cd0bd25c3c75f89 P c56a85c19c0841ec9899441418fd5ed2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c56a85c19c0841ec9899441418fd5ed2, State: Initialized, Role: FOLLOWER
I20240502 08:02:17.769666 7479 consensus_queue.cc:260] T bafc82a647b24cda8cd0bd25c3c75f89 P c56a85c19c0841ec9899441418fd5ed2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c56a85c19c0841ec9899441418fd5ed2" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38389 } }
I20240502 08:02:17.770443 7479 raft_consensus.cc:388] T bafc82a647b24cda8cd0bd25c3c75f89 P c56a85c19c0841ec9899441418fd5ed2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:02:17.770819 7479 raft_consensus.cc:482] T bafc82a647b24cda8cd0bd25c3c75f89 P c56a85c19c0841ec9899441418fd5ed2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:02:17.771211 7479 raft_consensus.cc:3037] T bafc82a647b24cda8cd0bd25c3c75f89 P c56a85c19c0841ec9899441418fd5ed2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:02:17.769815 7480 raft_consensus.cc:348] T 14b4fa2a60874e81a585c6e84ca1f7f1 P 8f72ded747a044928707b7e44be40025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f72ded747a044928707b7e44be40025" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 43287 } }
I20240502 08:02:17.771663 7480 raft_consensus.cc:374] T 14b4fa2a60874e81a585c6e84ca1f7f1 P 8f72ded747a044928707b7e44be40025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:02:17.772029 7480 raft_consensus.cc:729] T 14b4fa2a60874e81a585c6e84ca1f7f1 P 8f72ded747a044928707b7e44be40025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f72ded747a044928707b7e44be40025, State: Initialized, Role: FOLLOWER
I20240502 08:02:17.772888 7481 tablet_bootstrap.cc:492] T 8a46172374b5455c9dd94e6c44701f66 P 5d43c104f1ff4ef993228c4202aecc8e: No bootstrap required, opened a new log
I20240502 08:02:17.772653 7480 consensus_queue.cc:260] T 14b4fa2a60874e81a585c6e84ca1f7f1 P 8f72ded747a044928707b7e44be40025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f72ded747a044928707b7e44be40025" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 43287 } }
I20240502 08:02:17.773278 7481 ts_tablet_manager.cc:1367] T 8a46172374b5455c9dd94e6c44701f66 P 5d43c104f1ff4ef993228c4202aecc8e: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.009s
I20240502 08:02:17.773386 7480 raft_consensus.cc:388] T 14b4fa2a60874e81a585c6e84ca1f7f1 P 8f72ded747a044928707b7e44be40025 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:02:17.773766 7480 raft_consensus.cc:482] T 14b4fa2a60874e81a585c6e84ca1f7f1 P 8f72ded747a044928707b7e44be40025 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:02:17.774119 7480 raft_consensus.cc:3037] T 14b4fa2a60874e81a585c6e84ca1f7f1 P 8f72ded747a044928707b7e44be40025 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:02:17.775395 7481 raft_consensus.cc:348] T 8a46172374b5455c9dd94e6c44701f66 P 5d43c104f1ff4ef993228c4202aecc8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d43c104f1ff4ef993228c4202aecc8e" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40895 } }
I20240502 08:02:17.775836 7481 raft_consensus.cc:374] T 8a46172374b5455c9dd94e6c44701f66 P 5d43c104f1ff4ef993228c4202aecc8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:02:17.776122 7481 raft_consensus.cc:729] T 8a46172374b5455c9dd94e6c44701f66 P 5d43c104f1ff4ef993228c4202aecc8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d43c104f1ff4ef993228c4202aecc8e, State: Initialized, Role: FOLLOWER
I20240502 08:02:17.776664 7481 consensus_queue.cc:260] T 8a46172374b5455c9dd94e6c44701f66 P 5d43c104f1ff4ef993228c4202aecc8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d43c104f1ff4ef993228c4202aecc8e" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40895 } }
I20240502 08:02:17.777156 7481 raft_consensus.cc:388] T 8a46172374b5455c9dd94e6c44701f66 P 5d43c104f1ff4ef993228c4202aecc8e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:02:17.777390 7481 raft_consensus.cc:482] T 8a46172374b5455c9dd94e6c44701f66 P 5d43c104f1ff4ef993228c4202aecc8e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:02:17.777686 7481 raft_consensus.cc:3037] T 8a46172374b5455c9dd94e6c44701f66 P 5d43c104f1ff4ef993228c4202aecc8e [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:02:17.777949 7479 raft_consensus.cc:504] T bafc82a647b24cda8cd0bd25c3c75f89 P c56a85c19c0841ec9899441418fd5ed2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c56a85c19c0841ec9899441418fd5ed2" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38389 } }
I20240502 08:02:17.778707 7479 leader_election.cc:302] T bafc82a647b24cda8cd0bd25c3c75f89 P c56a85c19c0841ec9899441418fd5ed2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c56a85c19c0841ec9899441418fd5ed2; no voters:
I20240502 08:02:17.780818 7480 raft_consensus.cc:504] T 14b4fa2a60874e81a585c6e84ca1f7f1 P 8f72ded747a044928707b7e44be40025 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f72ded747a044928707b7e44be40025" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 43287 } }
I20240502 08:02:17.781847 7480 leader_election.cc:302] T 14b4fa2a60874e81a585c6e84ca1f7f1 P 8f72ded747a044928707b7e44be40025 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f72ded747a044928707b7e44be40025; no voters:
I20240502 08:02:17.783663 7481 raft_consensus.cc:504] T 8a46172374b5455c9dd94e6c44701f66 P 5d43c104f1ff4ef993228c4202aecc8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d43c104f1ff4ef993228c4202aecc8e" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40895 } }
I20240502 08:02:17.784404 7481 leader_election.cc:302] T 8a46172374b5455c9dd94e6c44701f66 P 5d43c104f1ff4ef993228c4202aecc8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d43c104f1ff4ef993228c4202aecc8e; no voters:
I20240502 08:02:17.784842 7479 leader_election.cc:288] T bafc82a647b24cda8cd0bd25c3c75f89 P c56a85c19c0841ec9899441418fd5ed2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:02:17.785444 7485 raft_consensus.cc:2781] T bafc82a647b24cda8cd0bd25c3c75f89 P c56a85c19c0841ec9899441418fd5ed2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:02:17.788875 7485 raft_consensus.cc:686] T bafc82a647b24cda8cd0bd25c3c75f89 P c56a85c19c0841ec9899441418fd5ed2 [term 1 LEADER]: Becoming Leader. State: Replica: c56a85c19c0841ec9899441418fd5ed2, State: Running, Role: LEADER
I20240502 08:02:17.789677 7487 raft_consensus.cc:2781] T 8a46172374b5455c9dd94e6c44701f66 P 5d43c104f1ff4ef993228c4202aecc8e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:02:17.789865 7485 consensus_queue.cc:237] T bafc82a647b24cda8cd0bd25c3c75f89 P c56a85c19c0841ec9899441418fd5ed2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c56a85c19c0841ec9899441418fd5ed2" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38389 } }
I20240502 08:02:17.791520 7481 leader_election.cc:288] T 8a46172374b5455c9dd94e6c44701f66 P 5d43c104f1ff4ef993228c4202aecc8e [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:02:17.794461 7480 leader_election.cc:288] T 14b4fa2a60874e81a585c6e84ca1f7f1 P 8f72ded747a044928707b7e44be40025 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:02:17.797848 7481 ts_tablet_manager.cc:1398] T 8a46172374b5455c9dd94e6c44701f66 P 5d43c104f1ff4ef993228c4202aecc8e: Time spent starting tablet: real 0.024s user 0.016s sys 0.004s
I20240502 08:02:17.798605 7479 ts_tablet_manager.cc:1398] T bafc82a647b24cda8cd0bd25c3c75f89 P c56a85c19c0841ec9899441418fd5ed2: Time spent starting tablet: real 0.033s user 0.014s sys 0.009s
I20240502 08:02:17.799827 7479 tablet_bootstrap.cc:492] T 5630b3f83a6b48249dfd0269ef496ffd P c56a85c19c0841ec9899441418fd5ed2: Bootstrap starting.
I20240502 08:02:17.804184 7486 raft_consensus.cc:2781] T 14b4fa2a60874e81a585c6e84ca1f7f1 P 8f72ded747a044928707b7e44be40025 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:02:17.796473 7487 raft_consensus.cc:686] T 8a46172374b5455c9dd94e6c44701f66 P 5d43c104f1ff4ef993228c4202aecc8e [term 1 LEADER]: Becoming Leader. State: Replica: 5d43c104f1ff4ef993228c4202aecc8e, State: Running, Role: LEADER
I20240502 08:02:17.806948 7487 consensus_queue.cc:237] T 8a46172374b5455c9dd94e6c44701f66 P 5d43c104f1ff4ef993228c4202aecc8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d43c104f1ff4ef993228c4202aecc8e" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40895 } }
I20240502 08:02:17.807626 7060 catalog_manager.cc:5484] T bafc82a647b24cda8cd0bd25c3c75f89 P c56a85c19c0841ec9899441418fd5ed2 reported cstate change: term changed from 0 to 1, leader changed from <none> to c56a85c19c0841ec9899441418fd5ed2 (127.6.204.193). New cstate: current_term: 1 leader_uuid: "c56a85c19c0841ec9899441418fd5ed2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c56a85c19c0841ec9899441418fd5ed2" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38389 } health_report { overall_health: HEALTHY } } }
I20240502 08:02:17.844277 7479 tablet_bootstrap.cc:654] T 5630b3f83a6b48249dfd0269ef496ffd P c56a85c19c0841ec9899441418fd5ed2: Neither blocks nor log segments found. Creating new log.
I20240502 08:02:17.879027 7486 raft_consensus.cc:686] T 14b4fa2a60874e81a585c6e84ca1f7f1 P 8f72ded747a044928707b7e44be40025 [term 1 LEADER]: Becoming Leader. State: Replica: 8f72ded747a044928707b7e44be40025, State: Running, Role: LEADER
I20240502 08:02:17.879871 7486 consensus_queue.cc:237] T 14b4fa2a60874e81a585c6e84ca1f7f1 P 8f72ded747a044928707b7e44be40025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f72ded747a044928707b7e44be40025" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 43287 } }
I20240502 08:02:17.880724 7480 ts_tablet_manager.cc:1398] T 14b4fa2a60874e81a585c6e84ca1f7f1 P 8f72ded747a044928707b7e44be40025: Time spent starting tablet: real 0.113s user 0.034s sys 0.056s
I20240502 08:02:17.893532 7060 catalog_manager.cc:5484] T 8a46172374b5455c9dd94e6c44701f66 P 5d43c104f1ff4ef993228c4202aecc8e reported cstate change: term changed from 0 to 1, leader changed from <none> to 5d43c104f1ff4ef993228c4202aecc8e (127.6.204.195). New cstate: current_term: 1 leader_uuid: "5d43c104f1ff4ef993228c4202aecc8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d43c104f1ff4ef993228c4202aecc8e" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40895 } health_report { overall_health: HEALTHY } } }
I20240502 08:02:17.908362 7479 tablet_bootstrap.cc:492] T 5630b3f83a6b48249dfd0269ef496ffd P c56a85c19c0841ec9899441418fd5ed2: No bootstrap required, opened a new log
I20240502 08:02:17.908903 7479 ts_tablet_manager.cc:1367] T 5630b3f83a6b48249dfd0269ef496ffd P c56a85c19c0841ec9899441418fd5ed2: Time spent bootstrapping tablet: real 0.109s user 0.014s sys 0.001s
I20240502 08:02:17.912626 7479 raft_consensus.cc:348] T 5630b3f83a6b48249dfd0269ef496ffd P c56a85c19c0841ec9899441418fd5ed2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c56a85c19c0841ec9899441418fd5ed2" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38389 } }
I20240502 08:02:17.913323 7479 raft_consensus.cc:374] T 5630b3f83a6b48249dfd0269ef496ffd P c56a85c19c0841ec9899441418fd5ed2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:02:17.913683 7479 raft_consensus.cc:729] T 5630b3f83a6b48249dfd0269ef496ffd P c56a85c19c0841ec9899441418fd5ed2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c56a85c19c0841ec9899441418fd5ed2, State: Initialized, Role: FOLLOWER
I20240502 08:02:17.914621 7479 consensus_queue.cc:260] T 5630b3f83a6b48249dfd0269ef496ffd P c56a85c19c0841ec9899441418fd5ed2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c56a85c19c0841ec9899441418fd5ed2" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38389 } }
I20240502 08:02:17.915380 7479 raft_consensus.cc:388] T 5630b3f83a6b48249dfd0269ef496ffd P c56a85c19c0841ec9899441418fd5ed2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:02:17.915742 7479 raft_consensus.cc:482] T 5630b3f83a6b48249dfd0269ef496ffd P c56a85c19c0841ec9899441418fd5ed2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:02:17.916162 7479 raft_consensus.cc:3037] T 5630b3f83a6b48249dfd0269ef496ffd P c56a85c19c0841ec9899441418fd5ed2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:02:17.919145 7058 catalog_manager.cc:5484] T 14b4fa2a60874e81a585c6e84ca1f7f1 P 8f72ded747a044928707b7e44be40025 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f72ded747a044928707b7e44be40025 (127.6.204.194). New cstate: current_term: 1 leader_uuid: "8f72ded747a044928707b7e44be40025" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f72ded747a044928707b7e44be40025" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 43287 } health_report { overall_health: HEALTHY } } }
I20240502 08:02:17.923245 7479 raft_consensus.cc:504] T 5630b3f83a6b48249dfd0269ef496ffd P c56a85c19c0841ec9899441418fd5ed2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c56a85c19c0841ec9899441418fd5ed2" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38389 } }
I20240502 08:02:17.924032 7479 leader_election.cc:302] T 5630b3f83a6b48249dfd0269ef496ffd P c56a85c19c0841ec9899441418fd5ed2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c56a85c19c0841ec9899441418fd5ed2; no voters:
I20240502 08:02:17.924793 7485 raft_consensus.cc:2781] T 5630b3f83a6b48249dfd0269ef496ffd P c56a85c19c0841ec9899441418fd5ed2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:02:17.925279 7485 raft_consensus.cc:686] T 5630b3f83a6b48249dfd0269ef496ffd P c56a85c19c0841ec9899441418fd5ed2 [term 1 LEADER]: Becoming Leader. State: Replica: c56a85c19c0841ec9899441418fd5ed2, State: Running, Role: LEADER
I20240502 08:02:17.925944 7485 consensus_queue.cc:237] T 5630b3f83a6b48249dfd0269ef496ffd P c56a85c19c0841ec9899441418fd5ed2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c56a85c19c0841ec9899441418fd5ed2" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38389 } }
I20240502 08:02:17.930689 7479 leader_election.cc:288] T 5630b3f83a6b48249dfd0269ef496ffd P c56a85c19c0841ec9899441418fd5ed2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:02:17.932444 7479 ts_tablet_manager.cc:1398] T 5630b3f83a6b48249dfd0269ef496ffd P c56a85c19c0841ec9899441418fd5ed2: Time spent starting tablet: real 0.023s user 0.010s sys 0.005s
I20240502 08:02:17.973991 7060 catalog_manager.cc:5484] T 5630b3f83a6b48249dfd0269ef496ffd P c56a85c19c0841ec9899441418fd5ed2 reported cstate change: term changed from 0 to 1, leader changed from <none> to c56a85c19c0841ec9899441418fd5ed2 (127.6.204.193). New cstate: current_term: 1 leader_uuid: "c56a85c19c0841ec9899441418fd5ed2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c56a85c19c0841ec9899441418fd5ed2" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38389 } health_report { overall_health: HEALTHY } } }
I20240502 08:02:18.014425 6963 tablet_server.cc:178] TabletServer@127.6.204.193:0 shutting down...
I20240502 08:02:18.033088 6963 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 08:02:18.033859 6963 tablet_replica.cc:330] T 5630b3f83a6b48249dfd0269ef496ffd P c56a85c19c0841ec9899441418fd5ed2: stopping tablet replica
I20240502 08:02:18.034541 6963 raft_consensus.cc:2229] T 5630b3f83a6b48249dfd0269ef496ffd P c56a85c19c0841ec9899441418fd5ed2 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:02:18.034984 6963 raft_consensus.cc:2258] T 5630b3f83a6b48249dfd0269ef496ffd P c56a85c19c0841ec9899441418fd5ed2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:02:18.037475 6963 tablet_replica.cc:330] T bafc82a647b24cda8cd0bd25c3c75f89 P c56a85c19c0841ec9899441418fd5ed2: stopping tablet replica
I20240502 08:02:18.037914 6963 raft_consensus.cc:2229] T bafc82a647b24cda8cd0bd25c3c75f89 P c56a85c19c0841ec9899441418fd5ed2 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:02:18.038314 6963 raft_consensus.cc:2258] T bafc82a647b24cda8cd0bd25c3c75f89 P c56a85c19c0841ec9899441418fd5ed2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:02:18.191020 6963 tablet_server.cc:194] TabletServer@127.6.204.193:0 shutdown complete.
I20240502 08:02:18.200232 6963 tablet_server.cc:178] TabletServer@127.6.204.194:0 shutting down...
I20240502 08:02:18.216087 6963 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 08:02:18.216885 6963 tablet_replica.cc:330] T 14b4fa2a60874e81a585c6e84ca1f7f1 P 8f72ded747a044928707b7e44be40025: stopping tablet replica
I20240502 08:02:18.217344 6963 raft_consensus.cc:2229] T 14b4fa2a60874e81a585c6e84ca1f7f1 P 8f72ded747a044928707b7e44be40025 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:02:18.217706 6963 raft_consensus.cc:2258] T 14b4fa2a60874e81a585c6e84ca1f7f1 P 8f72ded747a044928707b7e44be40025 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:02:18.237095 6963 tablet_server.cc:194] TabletServer@127.6.204.194:0 shutdown complete.
I20240502 08:02:18.244450 6963 tablet_server.cc:178] TabletServer@127.6.204.195:0 shutting down...
I20240502 08:02:18.259614 6963 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 08:02:18.260370 6963 tablet_replica.cc:330] T 8a46172374b5455c9dd94e6c44701f66 P 5d43c104f1ff4ef993228c4202aecc8e: stopping tablet replica
I20240502 08:02:18.260910 6963 raft_consensus.cc:2229] T 8a46172374b5455c9dd94e6c44701f66 P 5d43c104f1ff4ef993228c4202aecc8e [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:02:18.261371 6963 raft_consensus.cc:2258] T 8a46172374b5455c9dd94e6c44701f66 P 5d43c104f1ff4ef993228c4202aecc8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:02:18.271221 6963 tablet_server.cc:194] TabletServer@127.6.204.195:0 shutdown complete.
I20240502 08:02:18.278620 6963 master.cc:537] Master@127.6.204.254:35441 shutting down...
I20240502 08:02:18.376096 6963 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 5a6bf0a0ba2f44bb879ba72ea9378a12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:02:18.376641 6963 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 5a6bf0a0ba2f44bb879ba72ea9378a12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:02:18.376982 6963 tablet_replica.cc:330] T 00000000000000000000000000000000 P 5a6bf0a0ba2f44bb879ba72ea9378a12: stopping tablet replica
I20240502 08:02:18.395370 6963 master.cc:552] Master@127.6.204.254:35441 shutdown complete.
I20240502 08:02:18.429268 6963 master.cc:537] Master@127.6.204.253:42053 shutting down...
I20240502 08:02:18.441841 6963 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 40039c085f394e6c966aeee46f9425fb [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:02:18.442689 6963 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 40039c085f394e6c966aeee46f9425fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:02:18.443009 6963 tablet_replica.cc:330] T 00000000000000000000000000000000 P 40039c085f394e6c966aeee46f9425fb: stopping tablet replica
I20240502 08:02:18.495216 6963 master.cc:552] Master@127.6.204.253:42053 shutdown complete.
I20240502 08:02:18.508726 6963 master.cc:537] Master@127.6.204.252:45599 shutting down...
I20240502 08:02:18.521354 6963 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 62726479f252481b8f57e1dcba2d58b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:02:18.521816 6963 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 62726479f252481b8f57e1dcba2d58b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:02:18.522063 6963 tablet_replica.cc:330] T 00000000000000000000000000000000 P 62726479f252481b8f57e1dcba2d58b0: stopping tablet replica
I20240502 08:02:18.539273 6963 master.cc:552] Master@127.6.204.252:45599 shutdown complete.
[ OK ] AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing (4779 ms)
[ RUN ] AutoRebalancerTest.AutoRebalancingTurnOffAndOn
I20240502 08:02:18.564152 6963 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.6.204.254:40853
I20240502 08:02:18.565397 6963 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:02:18.570192 7505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:02:18.571887 7508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:02:18.571988 7506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:02:18.572343 6963 server_base.cc:1034] running on GCE node
I20240502 08:02:18.573137 6963 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:02:18.573307 6963 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:02:18.573447 6963 hybrid_clock.cc:648] HybridClock initialized: now 1714636938573429 us; error 0 us; skew 500 ppm
I20240502 08:02:18.573925 6963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:02:18.580593 6963 webserver.cc:460] Webserver started at http://127.6.204.254:46497/ using document root <none> and password file <none>
I20240502 08:02:18.581049 6963 fs_manager.cc:359] Metadata directory not provided
I20240502 08:02:18.581229 6963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:02:18.581475 6963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:02:18.582567 6963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714636933625983-6963-0/minicluster-data/master-0-root/instance:
uuid: "d63c312fa1b34251997988782a79379c"
format_stamp: "Formatted at 2024-05-02 08:02:18 on dist-test-slave-32xs"
I20240502 08:02:18.586648 6963 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240502 08:02:18.589578 7513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:02:18.590314 6963 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 08:02:18.590672 6963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714636933625983-6963-0/minicluster-data/master-0-root
uuid: "d63c312fa1b34251997988782a79379c"
format_stamp: "Formatted at 2024-05-02 08:02:18 on dist-test-slave-32xs"
I20240502 08:02:18.591238 6963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714636933625983-6963-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714636933625983-6963-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714636933625983-6963-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:02:18.609036 6963 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:02:18.610080 6963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:02:18.643672 6963 rpc_server.cc:307] RPC server started. Bound to: 127.6.204.254:40853
I20240502 08:02:18.643759 7564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.204.254:40853 every 8 connection(s)
I20240502 08:02:18.647228 7565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:02:18.656834 7565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d63c312fa1b34251997988782a79379c: Bootstrap starting.
I20240502 08:02:18.660363 7565 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d63c312fa1b34251997988782a79379c: Neither blocks nor log segments found. Creating new log.
I20240502 08:02:18.663815 7565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d63c312fa1b34251997988782a79379c: No bootstrap required, opened a new log
I20240502 08:02:18.665450 7565 raft_consensus.cc:348] T 00000000000000000000000000000000 P d63c312fa1b34251997988782a79379c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d63c312fa1b34251997988782a79379c" member_type: VOTER }
I20240502 08:02:18.665871 7565 raft_consensus.cc:374] T 00000000000000000000000000000000 P d63c312fa1b34251997988782a79379c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:02:18.666065 7565 raft_consensus.cc:729] T 00000000000000000000000000000000 P d63c312fa1b34251997988782a79379c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d63c312fa1b34251997988782a79379c, State: Initialized, Role: FOLLOWER
I20240502 08:02:18.666538 7565 consensus_queue.cc:260] T 00000000000000000000000000000000 P d63c312fa1b34251997988782a79379c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d63c312fa1b34251997988782a79379c" member_type: VOTER }
I20240502 08:02:18.666937 7565 raft_consensus.cc:388] T 00000000000000000000000000000000 P d63c312fa1b34251997988782a79379c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:02:18.667117 7565 raft_consensus.cc:482] T 00000000000000000000000000000000 P d63c312fa1b34251997988782a79379c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:02:18.667318 7565 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d63c312fa1b34251997988782a79379c [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:02:18.670938 7565 raft_consensus.cc:504] T 00000000000000000000000000000000 P d63c312fa1b34251997988782a79379c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d63c312fa1b34251997988782a79379c" member_type: VOTER }
I20240502 08:02:18.671387 7565 leader_election.cc:302] T 00000000000000000000000000000000 P d63c312fa1b34251997988782a79379c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d63c312fa1b34251997988782a79379c; no voters:
I20240502 08:02:18.672391 7565 leader_election.cc:288] T 00000000000000000000000000000000 P d63c312fa1b34251997988782a79379c [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:02:18.672670 7568 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d63c312fa1b34251997988782a79379c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:02:18.673916 7568 raft_consensus.cc:686] T 00000000000000000000000000000000 P d63c312fa1b34251997988782a79379c [term 1 LEADER]: Becoming Leader. State: Replica: d63c312fa1b34251997988782a79379c, State: Running, Role: LEADER
I20240502 08:02:18.674484 7568 consensus_queue.cc:237] T 00000000000000000000000000000000 P d63c312fa1b34251997988782a79379c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d63c312fa1b34251997988782a79379c" member_type: VOTER }
I20240502 08:02:18.675117 7565 sys_catalog.cc:564] T 00000000000000000000000000000000 P d63c312fa1b34251997988782a79379c [sys.catalog]: configured and running, proceeding with master startup.
I20240502 08:02:18.677264 7569 sys_catalog.cc:455] T 00000000000000000000000000000000 P d63c312fa1b34251997988782a79379c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d63c312fa1b34251997988782a79379c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d63c312fa1b34251997988782a79379c" member_type: VOTER } }
I20240502 08:02:18.677358 7570 sys_catalog.cc:455] T 00000000000000000000000000000000 P d63c312fa1b34251997988782a79379c [sys.catalog]: SysCatalogTable state changed. Reason: New leader d63c312fa1b34251997988782a79379c. Latest consensus state: current_term: 1 leader_uuid: "d63c312fa1b34251997988782a79379c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d63c312fa1b34251997988782a79379c" member_type: VOTER } }
I20240502 08:02:18.677985 7570 sys_catalog.cc:458] T 00000000000000000000000000000000 P d63c312fa1b34251997988782a79379c [sys.catalog]: This master's current role is: LEADER
I20240502 08:02:18.678050 7569 sys_catalog.cc:458] T 00000000000000000000000000000000 P d63c312fa1b34251997988782a79379c [sys.catalog]: This master's current role is: LEADER
I20240502 08:02:18.682339 7573 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 08:02:18.687326 7573 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 08:02:18.688939 6963 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 08:02:18.695353 7573 catalog_manager.cc:1347] Generated new cluster ID: 7792a8bf0fd44e9a97596f65f849b768
I20240502 08:02:18.695586 7573 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 08:02:18.715458 7573 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 08:02:18.716606 7573 catalog_manager.cc:1504] Loading token signing keys...
I20240502 08:02:18.731453 7573 catalog_manager.cc:5857] T 00000000000000000000000000000000 P d63c312fa1b34251997988782a79379c: Generated new TSK 0
I20240502 08:02:18.732012 7573 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 08:02:18.755203 6963 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:02:18.760401 7586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:02:18.761593 7587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:02:18.762916 7589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:02:18.765676 6963 server_base.cc:1034] running on GCE node
I20240502 08:02:18.768071 6963 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:02:18.768252 6963 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:02:18.768412 6963 hybrid_clock.cc:648] HybridClock initialized: now 1714636938768395 us; error 0 us; skew 500 ppm
I20240502 08:02:18.768913 6963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:02:18.771247 6963 webserver.cc:460] Webserver started at http://127.6.204.193:38417/ using document root <none> and password file <none>
I20240502 08:02:18.771694 6963 fs_manager.cc:359] Metadata directory not provided
I20240502 08:02:18.771869 6963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:02:18.772114 6963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:02:18.773144 6963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714636933625983-6963-0/minicluster-data/ts-0-root/instance:
uuid: "d5cb06cedbcb4941afcceab604f34ea4"
format_stamp: "Formatted at 2024-05-02 08:02:18 on dist-test-slave-32xs"
I20240502 08:02:18.777550 6963 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240502 08:02:18.780661 7594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:02:18.781466 6963 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 08:02:18.781822 6963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714636933625983-6963-0/minicluster-data/ts-0-root
uuid: "d5cb06cedbcb4941afcceab604f34ea4"
format_stamp: "Formatted at 2024-05-02 08:02:18 on dist-test-slave-32xs"
I20240502 08:02:18.782105 6963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714636933625983-6963-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714636933625983-6963-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714636933625983-6963-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:02:18.802896 6963 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:02:18.803870 6963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:02:18.805213 6963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:02:18.807410 6963 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:02:18.807596 6963 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:02:18.807816 6963 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:02:18.807965 6963 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:02:18.844677 6963 rpc_server.cc:307] RPC server started. Bound to: 127.6.204.193:42525
I20240502 08:02:18.844764 7656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.204.193:42525 every 8 connection(s)
I20240502 08:02:18.849012 6963 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:02:18.856118 7661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:02:18.858659 7662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:02:18.860113 7664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:02:18.860920 6963 server_base.cc:1034] running on GCE node
I20240502 08:02:18.861793 6963 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:02:18.862010 6963 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:02:18.862238 6963 hybrid_clock.cc:648] HybridClock initialized: now 1714636938862221 us; error 0 us; skew 500 ppm
I20240502 08:02:18.862774 6963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:02:18.863195 7657 heartbeater.cc:347] Connected to a master server at 127.6.204.254:40853
I20240502 08:02:18.863528 7657 heartbeater.cc:464] Registering TS with master...
I20240502 08:02:18.864461 7657 heartbeater.cc:511] Master 127.6.204.254:40853 requested a full tablet report, sending...
I20240502 08:02:18.865620 6963 webserver.cc:460] Webserver started at http://127.6.204.194:44839/ using document root <none> and password file <none>
I20240502 08:02:18.866232 6963 fs_manager.cc:359] Metadata directory not provided
I20240502 08:02:18.866411 6963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:02:18.866667 6963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:02:18.866649 7530 ts_manager.cc:194] Registered new tserver with Master: d5cb06cedbcb4941afcceab604f34ea4 (127.6.204.193:42525)
I20240502 08:02:18.867878 6963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714636933625983-6963-0/minicluster-data/ts-1-root/instance:
uuid: "16eac9aafbee42bd8a83544ec9b97e74"
format_stamp: "Formatted at 2024-05-02 08:02:18 on dist-test-slave-32xs"
I20240502 08:02:18.868508 7530 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59996
I20240502 08:02:18.872534 6963 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 08:02:18.875691 7669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:02:18.876430 6963 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 08:02:18.876695 6963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714636933625983-6963-0/minicluster-data/ts-1-root
uuid: "16eac9aafbee42bd8a83544ec9b97e74"
format_stamp: "Formatted at 2024-05-02 08:02:18 on dist-test-slave-32xs"
I20240502 08:02:18.876989 6963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714636933625983-6963-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714636933625983-6963-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714636933625983-6963-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:02:18.896700 6963 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:02:18.897696 6963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:02:18.899122 6963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:02:18.901291 6963 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:02:18.901468 6963 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:02:18.901710 6963 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:02:18.901854 6963 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:02:18.938205 6963 rpc_server.cc:307] RPC server started. Bound to: 127.6.204.194:38399
I20240502 08:02:18.938284 7731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.204.194:38399 every 8 connection(s)
I20240502 08:02:18.942786 6963 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:02:18.950405 7735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:02:18.950732 7736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:02:18.953547 7732 heartbeater.cc:347] Connected to a master server at 127.6.204.254:40853
I20240502 08:02:18.954042 7732 heartbeater.cc:464] Registering TS with master...
I20240502 08:02:18.955018 7732 heartbeater.cc:511] Master 127.6.204.254:40853 requested a full tablet report, sending...
W20240502 08:02:18.955263 7738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:02:18.955330 6963 server_base.cc:1034] running on GCE node
I20240502 08:02:18.956352 6963 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:02:18.956568 6963 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:02:18.956734 6963 hybrid_clock.cc:648] HybridClock initialized: now 1714636938956719 us; error 0 us; skew 500 ppm
I20240502 08:02:18.957226 6963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:02:18.957191 7530 ts_manager.cc:194] Registered new tserver with Master: 16eac9aafbee42bd8a83544ec9b97e74 (127.6.204.194:38399)
I20240502 08:02:18.959029 7530 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60012
I20240502 08:02:18.960026 6963 webserver.cc:460] Webserver started at http://127.6.204.195:43077/ using document root <none> and password file <none>
I20240502 08:02:18.960495 6963 fs_manager.cc:359] Metadata directory not provided
I20240502 08:02:18.960686 6963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:02:18.960932 6963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:02:18.961957 6963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714636933625983-6963-0/minicluster-data/ts-2-root/instance:
uuid: "36331454fb984af5a6d4f35efe0a2b00"
format_stamp: "Formatted at 2024-05-02 08:02:18 on dist-test-slave-32xs"
I20240502 08:02:18.965991 6963 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20240502 08:02:18.968962 7743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:02:18.969691 6963 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 08:02:18.969964 6963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714636933625983-6963-0/minicluster-data/ts-2-root
uuid: "36331454fb984af5a6d4f35efe0a2b00"
format_stamp: "Formatted at 2024-05-02 08:02:18 on dist-test-slave-32xs"
I20240502 08:02:18.970283 6963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714636933625983-6963-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714636933625983-6963-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714636933625983-6963-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:02:19.009122 6963 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:02:19.010179 6963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:02:19.011605 6963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:02:19.013775 6963 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:02:19.013968 6963 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:02:19.014209 6963 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:02:19.014364 6963 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:02:19.050772 6963 rpc_server.cc:307] RPC server started. Bound to: 127.6.204.195:40297
I20240502 08:02:19.050863 7805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.204.195:40297 every 8 connection(s)
I20240502 08:02:19.063431 7806 heartbeater.cc:347] Connected to a master server at 127.6.204.254:40853
I20240502 08:02:19.063809 7806 heartbeater.cc:464] Registering TS with master...
I20240502 08:02:19.064507 7806 heartbeater.cc:511] Master 127.6.204.254:40853 requested a full tablet report, sending...
I20240502 08:02:19.066277 7530 ts_manager.cc:194] Registered new tserver with Master: 36331454fb984af5a6d4f35efe0a2b00 (127.6.204.195:40297)
I20240502 08:02:19.067379 6963 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013553028s
I20240502 08:02:19.067605 7530 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60024
I20240502 08:02:19.871150 7657 heartbeater.cc:503] Master 127.6.204.254:40853 was elected leader, sending a full tablet report...
I20240502 08:02:19.961478 7732 heartbeater.cc:503] Master 127.6.204.254:40853 was elected leader, sending a full tablet report...
I20240502 08:02:20.070309 7806 heartbeater.cc:503] Master 127.6.204.254:40853 was elected leader, sending a full tablet report...
I20240502 08:02:20.099835 6963 test_util.cc:268] Using random seed: 96190691
I20240502 08:02:20.121789 7530 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48264:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\017\004\001\000\376\377\377\037\004\001\000\375\377\377/\004\001\000\374\377\377?\004\001\000\373\377\377O\004\001\000\372\377\377_\004\001\000\371\377\377o""\004\001\000\377\377\377\017\004\001\000\376\377\377\037\004\001\000\375\377\377/\004\001\000\374\377\377?\004\001\000\373\377\377O\004\001\000\372\377\377_\004\001\000\371\377\377o"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:02:20.124230 7530 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:02:20.205647 7771 tablet_service.cc:1467] Processing CreateTablet for tablet d4042d449ac74cb6bbeb7ba7f90131c4 (DEFAULT_TABLE table=test-workload [id=dcc022a188494bf2bf8faf28f01e2cca]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240502 08:02:20.205912 7622 tablet_service.cc:1467] Processing CreateTablet for tablet d4042d449ac74cb6bbeb7ba7f90131c4 (DEFAULT_TABLE table=test-workload [id=dcc022a188494bf2bf8faf28f01e2cca]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240502 08:02:20.207546 7771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4042d449ac74cb6bbeb7ba7f90131c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:02:20.207829 7620 tablet_service.cc:1467] Processing CreateTablet for tablet ac59612a1fa642f6b7dfd4d899bd6f9d (DEFAULT_TABLE table=test-workload [id=dcc022a188494bf2bf8faf28f01e2cca]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240502 08:02:20.207864 7622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4042d449ac74cb6bbeb7ba7f90131c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:02:20.209178 7620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac59612a1fa642f6b7dfd4d899bd6f9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:02:20.210394 7621 tablet_service.cc:1467] Processing CreateTablet for tablet cb9b5608afa2415e905426d7400abd6c (DEFAULT_TABLE table=test-workload [id=dcc022a188494bf2bf8faf28f01e2cca]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240502 08:02:20.211151 7769 tablet_service.cc:1467] Processing CreateTablet for tablet ac59612a1fa642f6b7dfd4d899bd6f9d (DEFAULT_TABLE table=test-workload [id=dcc022a188494bf2bf8faf28f01e2cca]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240502 08:02:20.211722 7621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb9b5608afa2415e905426d7400abd6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:02:20.212456 7769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac59612a1fa642f6b7dfd4d899bd6f9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:02:20.215462 7768 tablet_service.cc:1467] Processing CreateTablet for tablet af8197b1de49493a966b8ba1146ada68 (DEFAULT_TABLE table=test-workload [id=dcc022a188494bf2bf8faf28f01e2cca]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240502 08:02:20.216238 7770 tablet_service.cc:1467] Processing CreateTablet for tablet cb9b5608afa2415e905426d7400abd6c (DEFAULT_TABLE table=test-workload [id=dcc022a188494bf2bf8faf28f01e2cca]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240502 08:02:20.216832 7768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af8197b1de49493a966b8ba1146ada68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:02:20.217514 7770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb9b5608afa2415e905426d7400abd6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:02:20.218895 7764 tablet_service.cc:1467] Processing CreateTablet for tablet f8a4589c0a9b417188a2d884011b0240 (DEFAULT_TABLE table=test-workload [id=dcc022a188494bf2bf8faf28f01e2cca]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240502 08:02:20.219079 7767 tablet_service.cc:1467] Processing CreateTablet for tablet 8f45c7e5e5cf4b4a84127573dd446bde (DEFAULT_TABLE table=test-workload [id=dcc022a188494bf2bf8faf28f01e2cca]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240502 08:02:20.220173 7764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8a4589c0a9b417188a2d884011b0240. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:02:20.221422 7766 tablet_service.cc:1467] Processing CreateTablet for tablet 8e0f7109a1a94eb999a919ed98dea1ce (DEFAULT_TABLE table=test-workload [id=dcc022a188494bf2bf8faf28f01e2cca]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240502 08:02:20.222733 7766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e0f7109a1a94eb999a919ed98dea1ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:02:20.223847 7618 tablet_service.cc:1467] Processing CreateTablet for tablet 8f45c7e5e5cf4b4a84127573dd446bde (DEFAULT_TABLE table=test-workload [id=dcc022a188494bf2bf8faf28f01e2cca]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240502 08:02:20.224658 7765 tablet_service.cc:1467] Processing CreateTablet for tablet b7961f100fcd49f581e2b1bfa1f1ec05 (DEFAULT_TABLE table=test-workload [id=dcc022a188494bf2bf8faf28f01e2cca]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240502 08:02:20.225243 7618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f45c7e5e5cf4b4a84127573dd446bde. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:02:20.227373 7615 tablet_service.cc:1467] Processing CreateTablet for tablet f8a4589c0a9b417188a2d884011b0240 (DEFAULT_TABLE table=test-workload [id=dcc022a188494bf2bf8faf28f01e2cca]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240502 08:02:20.228677 7615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8a4589c0a9b417188a2d884011b0240. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:02:20.231355 7619 tablet_service.cc:1467] Processing CreateTablet for tablet af8197b1de49493a966b8ba1146ada68 (DEFAULT_TABLE table=test-workload [id=dcc022a188494bf2bf8faf28f01e2cca]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240502 08:02:20.231349 7617 tablet_service.cc:1467] Processing CreateTablet for tablet 8e0f7109a1a94eb999a919ed98dea1ce (DEFAULT_TABLE table=test-workload [id=dcc022a188494bf2bf8faf28f01e2cca]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240502 08:02:20.233045 7619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af8197b1de49493a966b8ba1146ada68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:02:20.233640 7616 tablet_service.cc:1467] Processing CreateTablet for tablet b7961f100fcd49f581e2b1bfa1f1ec05 (DEFAULT_TABLE table=test-workload [id=dcc022a188494bf2bf8faf28f01e2cca]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240502 08:02:20.234921 7616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7961f100fcd49f581e2b1bfa1f1ec05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:02:20.238662 7617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e0f7109a1a94eb999a919ed98dea1ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:02:20.226008 7765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7961f100fcd49f581e2b1bfa1f1ec05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:02:20.242547 7767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f45c7e5e5cf4b4a84127573dd446bde. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:02:20.252231 7697 tablet_service.cc:1467] Processing CreateTablet for tablet d4042d449ac74cb6bbeb7ba7f90131c4 (DEFAULT_TABLE table=test-workload [id=dcc022a188494bf2bf8faf28f01e2cca]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240502 08:02:20.254101 7697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4042d449ac74cb6bbeb7ba7f90131c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:02:20.264811 7696 tablet_service.cc:1467] Processing CreateTablet for tablet cb9b5608afa2415e905426d7400abd6c (DEFAULT_TABLE table=test-workload [id=dcc022a188494bf2bf8faf28f01e2cca]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240502 08:02:20.266893 7696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb9b5608afa2415e905426d7400abd6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:02:20.272965 7695 tablet_service.cc:1467] Processing CreateTablet for tablet ac59612a1fa642f6b7dfd4d899bd6f9d (DEFAULT_TABLE table=test-workload [id=dcc022a188494bf2bf8faf28f01e2cca]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240502 08:02:20.274420 7695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac59612a1fa642f6b7dfd4d899bd6f9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:02:20.276144 7694 tablet_service.cc:1467] Processing CreateTablet for tablet af8197b1de49493a966b8ba1146ada68 (DEFAULT_TABLE table=test-workload [id=dcc022a188494bf2bf8faf28f01e2cca]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240502 08:02:20.278028 7694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af8197b1de49493a966b8ba1146ada68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:02:20.296625 7693 tablet_service.cc:1467] Processing CreateTablet for tablet 8f45c7e5e5cf4b4a84127573dd446bde (DEFAULT_TABLE table=test-workload [id=dcc022a188494bf2bf8faf28f01e2cca]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240502 08:02:20.299065 7693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f45c7e5e5cf4b4a84127573dd446bde. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:02:20.301285 7690 tablet_service.cc:1467] Processing CreateTablet for tablet f8a4589c0a9b417188a2d884011b0240 (DEFAULT_TABLE table=test-workload [id=dcc022a188494bf2bf8faf28f01e2cca]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240502 08:02:20.306551 7826 tablet_bootstrap.cc:492] T d4042d449ac74cb6bbeb7ba7f90131c4 P 36331454fb984af5a6d4f35efe0a2b00: Bootstrap starting.
I20240502 08:02:20.306656 7690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8a4589c0a9b417188a2d884011b0240. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:02:20.304776 7692 tablet_service.cc:1467] Processing CreateTablet for tablet 8e0f7109a1a94eb999a919ed98dea1ce (DEFAULT_TABLE table=test-workload [id=dcc022a188494bf2bf8faf28f01e2cca]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240502 08:02:20.313477 7691 tablet_service.cc:1467] Processing CreateTablet for tablet b7961f100fcd49f581e2b1bfa1f1ec05 (DEFAULT_TABLE table=test-workload [id=dcc022a188494bf2bf8faf28f01e2cca]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240502 08:02:20.315356 7691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7961f100fcd49f581e2b1bfa1f1ec05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:02:20.317610 7692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e0f7109a1a94eb999a919ed98dea1ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:02:20.329929 7827 tablet_bootstrap.cc:492] T d4042d449ac74cb6bbeb7ba7f90131c4 P d5cb06cedbcb4941afcceab604f34ea4: Bootstrap starting.
I20240502 08:02:20.341487 7827 tablet_bootstrap.cc:654] T d4042d449ac74cb6bbeb7ba7f90131c4 P d5cb06cedbcb4941afcceab604f34ea4: Neither blocks nor log segments found. Creating new log.
I20240502 08:02:20.362946 7827 tablet_bootstrap.cc:492] T d4042d449ac74cb6bbeb7ba7f90131c4 P d5cb06cedbcb4941afcceab604f34ea4: No bootstrap required, opened a new log
I20240502 08:02:20.365136 7827 ts_tablet_manager.cc:1367] T d4042d449ac74cb6bbeb7ba7f90131c4 P d5cb06cedbcb4941afcceab604f34ea4: Time spent bootstrapping tablet: real 0.036s user 0.006s sys 0.006s
I20240502 08:02:20.366979 7826 tablet_bootstrap.cc:654] T d4042d449ac74cb6bbeb7ba7f90131c4 P 36331454fb984af5a6d4f35efe0a2b00: Neither blocks nor log segments found. Creating new log.
I20240502 08:02:20.368131 7827 raft_consensus.cc:348] T d4042d449ac74cb6bbeb7ba7f90131c4 P d5cb06cedbcb4941afcceab604f34ea4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } } peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } }
I20240502 08:02:20.372844 7827 raft_consensus.cc:374] T d4042d449ac74cb6bbeb7ba7f90131c4 P d5cb06cedbcb4941afcceab604f34ea4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:02:20.373217 7827 raft_consensus.cc:729] T d4042d449ac74cb6bbeb7ba7f90131c4 P d5cb06cedbcb4941afcceab604f34ea4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5cb06cedbcb4941afcceab604f34ea4, State: Initialized, Role: FOLLOWER
I20240502 08:02:20.374157 7827 consensus_queue.cc:260] T d4042d449ac74cb6bbeb7ba7f90131c4 P d5cb06cedbcb4941afcceab604f34ea4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } } peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } }
I20240502 08:02:20.389058 7826 tablet_bootstrap.cc:492] T d4042d449ac74cb6bbeb7ba7f90131c4 P 36331454fb984af5a6d4f35efe0a2b00: No bootstrap required, opened a new log
I20240502 08:02:20.389825 7826 ts_tablet_manager.cc:1367] T d4042d449ac74cb6bbeb7ba7f90131c4 P 36331454fb984af5a6d4f35efe0a2b00: Time spent bootstrapping tablet: real 0.084s user 0.011s sys 0.012s
I20240502 08:02:20.390247 7831 tablet_bootstrap.cc:492] T ac59612a1fa642f6b7dfd4d899bd6f9d P 16eac9aafbee42bd8a83544ec9b97e74: Bootstrap starting.
I20240502 08:02:20.393546 7826 raft_consensus.cc:348] T d4042d449ac74cb6bbeb7ba7f90131c4 P 36331454fb984af5a6d4f35efe0a2b00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } } peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } }
I20240502 08:02:20.394757 7826 raft_consensus.cc:374] T d4042d449ac74cb6bbeb7ba7f90131c4 P 36331454fb984af5a6d4f35efe0a2b00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:02:20.395004 7831 tablet_bootstrap.cc:654] T ac59612a1fa642f6b7dfd4d899bd6f9d P 16eac9aafbee42bd8a83544ec9b97e74: Neither blocks nor log segments found. Creating new log.
I20240502 08:02:20.395321 7826 raft_consensus.cc:729] T d4042d449ac74cb6bbeb7ba7f90131c4 P 36331454fb984af5a6d4f35efe0a2b00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36331454fb984af5a6d4f35efe0a2b00, State: Initialized, Role: FOLLOWER
I20240502 08:02:20.396482 7826 consensus_queue.cc:260] T d4042d449ac74cb6bbeb7ba7f90131c4 P 36331454fb984af5a6d4f35efe0a2b00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } } peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } }
I20240502 08:02:20.399129 7827 ts_tablet_manager.cc:1398] T d4042d449ac74cb6bbeb7ba7f90131c4 P d5cb06cedbcb4941afcceab604f34ea4: Time spent starting tablet: real 0.034s user 0.016s sys 0.013s
I20240502 08:02:20.400065 7826 ts_tablet_manager.cc:1398] T d4042d449ac74cb6bbeb7ba7f90131c4 P 36331454fb984af5a6d4f35efe0a2b00: Time spent starting tablet: real 0.010s user 0.006s sys 0.003s
I20240502 08:02:20.400210 7827 tablet_bootstrap.cc:492] T af8197b1de49493a966b8ba1146ada68 P d5cb06cedbcb4941afcceab604f34ea4: Bootstrap starting.
I20240502 08:02:20.400656 7831 tablet_bootstrap.cc:492] T ac59612a1fa642f6b7dfd4d899bd6f9d P 16eac9aafbee42bd8a83544ec9b97e74: No bootstrap required, opened a new log
I20240502 08:02:20.401186 7831 ts_tablet_manager.cc:1367] T ac59612a1fa642f6b7dfd4d899bd6f9d P 16eac9aafbee42bd8a83544ec9b97e74: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.005s
I20240502 08:02:20.403715 7831 raft_consensus.cc:348] T ac59612a1fa642f6b7dfd4d899bd6f9d P 16eac9aafbee42bd8a83544ec9b97e74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } }
I20240502 08:02:20.404493 7831 raft_consensus.cc:374] T ac59612a1fa642f6b7dfd4d899bd6f9d P 16eac9aafbee42bd8a83544ec9b97e74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:02:20.404781 7831 raft_consensus.cc:729] T ac59612a1fa642f6b7dfd4d899bd6f9d P 16eac9aafbee42bd8a83544ec9b97e74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16eac9aafbee42bd8a83544ec9b97e74, State: Initialized, Role: FOLLOWER
I20240502 08:02:20.405462 7831 consensus_queue.cc:260] T ac59612a1fa642f6b7dfd4d899bd6f9d P 16eac9aafbee42bd8a83544ec9b97e74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } }
I20240502 08:02:20.405997 7827 tablet_bootstrap.cc:654] T af8197b1de49493a966b8ba1146ada68 P d5cb06cedbcb4941afcceab604f34ea4: Neither blocks nor log segments found. Creating new log.
I20240502 08:02:20.407600 7826 tablet_bootstrap.cc:492] T cb9b5608afa2415e905426d7400abd6c P 36331454fb984af5a6d4f35efe0a2b00: Bootstrap starting.
I20240502 08:02:20.416121 7831 ts_tablet_manager.cc:1398] T ac59612a1fa642f6b7dfd4d899bd6f9d P 16eac9aafbee42bd8a83544ec9b97e74: Time spent starting tablet: real 0.015s user 0.007s sys 0.007s
I20240502 08:02:20.417088 7831 tablet_bootstrap.cc:492] T cb9b5608afa2415e905426d7400abd6c P 16eac9aafbee42bd8a83544ec9b97e74: Bootstrap starting.
I20240502 08:02:20.419840 7826 tablet_bootstrap.cc:654] T cb9b5608afa2415e905426d7400abd6c P 36331454fb984af5a6d4f35efe0a2b00: Neither blocks nor log segments found. Creating new log.
I20240502 08:02:20.422376 7831 tablet_bootstrap.cc:654] T cb9b5608afa2415e905426d7400abd6c P 16eac9aafbee42bd8a83544ec9b97e74: Neither blocks nor log segments found. Creating new log.
I20240502 08:02:20.427876 7827 tablet_bootstrap.cc:492] T af8197b1de49493a966b8ba1146ada68 P d5cb06cedbcb4941afcceab604f34ea4: No bootstrap required, opened a new log
I20240502 08:02:20.428411 7827 ts_tablet_manager.cc:1367] T af8197b1de49493a966b8ba1146ada68 P d5cb06cedbcb4941afcceab604f34ea4: Time spent bootstrapping tablet: real 0.028s user 0.017s sys 0.008s
I20240502 08:02:20.430296 7826 tablet_bootstrap.cc:492] T cb9b5608afa2415e905426d7400abd6c P 36331454fb984af5a6d4f35efe0a2b00: No bootstrap required, opened a new log
I20240502 08:02:20.430842 7826 ts_tablet_manager.cc:1367] T cb9b5608afa2415e905426d7400abd6c P 36331454fb984af5a6d4f35efe0a2b00: Time spent bootstrapping tablet: real 0.023s user 0.009s sys 0.006s
I20240502 08:02:20.431015 7827 raft_consensus.cc:348] T af8197b1de49493a966b8ba1146ada68 P d5cb06cedbcb4941afcceab604f34ea4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } }
I20240502 08:02:20.431824 7827 raft_consensus.cc:374] T af8197b1de49493a966b8ba1146ada68 P d5cb06cedbcb4941afcceab604f34ea4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:02:20.432125 7827 raft_consensus.cc:729] T af8197b1de49493a966b8ba1146ada68 P d5cb06cedbcb4941afcceab604f34ea4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5cb06cedbcb4941afcceab604f34ea4, State: Initialized, Role: FOLLOWER
I20240502 08:02:20.432924 7827 consensus_queue.cc:260] T af8197b1de49493a966b8ba1146ada68 P d5cb06cedbcb4941afcceab604f34ea4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } }
I20240502 08:02:20.433604 7831 tablet_bootstrap.cc:492] T cb9b5608afa2415e905426d7400abd6c P 16eac9aafbee42bd8a83544ec9b97e74: No bootstrap required, opened a new log
I20240502 08:02:20.434337 7831 ts_tablet_manager.cc:1367] T cb9b5608afa2415e905426d7400abd6c P 16eac9aafbee42bd8a83544ec9b97e74: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.011s
I20240502 08:02:20.433928 7826 raft_consensus.cc:348] T cb9b5608afa2415e905426d7400abd6c P 36331454fb984af5a6d4f35efe0a2b00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } }
I20240502 08:02:20.435174 7827 ts_tablet_manager.cc:1398] T af8197b1de49493a966b8ba1146ada68 P d5cb06cedbcb4941afcceab604f34ea4: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20240502 08:02:20.434715 7826 raft_consensus.cc:374] T cb9b5608afa2415e905426d7400abd6c P 36331454fb984af5a6d4f35efe0a2b00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:02:20.436111 7827 tablet_bootstrap.cc:492] T 8e0f7109a1a94eb999a919ed98dea1ce P d5cb06cedbcb4941afcceab604f34ea4: Bootstrap starting.
I20240502 08:02:20.436317 7826 raft_consensus.cc:729] T cb9b5608afa2415e905426d7400abd6c P 36331454fb984af5a6d4f35efe0a2b00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36331454fb984af5a6d4f35efe0a2b00, State: Initialized, Role: FOLLOWER
I20240502 08:02:20.436859 7831 raft_consensus.cc:348] T cb9b5608afa2415e905426d7400abd6c P 16eac9aafbee42bd8a83544ec9b97e74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } }
I20240502 08:02:20.437611 7831 raft_consensus.cc:374] T cb9b5608afa2415e905426d7400abd6c P 16eac9aafbee42bd8a83544ec9b97e74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:02:20.437155 7826 consensus_queue.cc:260] T cb9b5608afa2415e905426d7400abd6c P 36331454fb984af5a6d4f35efe0a2b00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } }
I20240502 08:02:20.437947 7831 raft_consensus.cc:729] T cb9b5608afa2415e905426d7400abd6c P 16eac9aafbee42bd8a83544ec9b97e74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16eac9aafbee42bd8a83544ec9b97e74, State: Initialized, Role: FOLLOWER
I20240502 08:02:20.438908 7831 consensus_queue.cc:260] T cb9b5608afa2415e905426d7400abd6c P 16eac9aafbee42bd8a83544ec9b97e74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } }
I20240502 08:02:20.441087 7831 ts_tablet_manager.cc:1398] T cb9b5608afa2415e905426d7400abd6c P 16eac9aafbee42bd8a83544ec9b97e74: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240502 08:02:20.441949 7827 tablet_bootstrap.cc:654] T 8e0f7109a1a94eb999a919ed98dea1ce P d5cb06cedbcb4941afcceab604f34ea4: Neither blocks nor log segments found. Creating new log.
I20240502 08:02:20.442047 7831 tablet_bootstrap.cc:492] T d4042d449ac74cb6bbeb7ba7f90131c4 P 16eac9aafbee42bd8a83544ec9b97e74: Bootstrap starting.
I20240502 08:02:20.446767 7826 ts_tablet_manager.cc:1398] T cb9b5608afa2415e905426d7400abd6c P 36331454fb984af5a6d4f35efe0a2b00: Time spent starting tablet: real 0.016s user 0.004s sys 0.003s
I20240502 08:02:20.448905 7831 tablet_bootstrap.cc:654] T d4042d449ac74cb6bbeb7ba7f90131c4 P 16eac9aafbee42bd8a83544ec9b97e74: Neither blocks nor log segments found. Creating new log.
I20240502 08:02:20.452490 7826 tablet_bootstrap.cc:492] T ac59612a1fa642f6b7dfd4d899bd6f9d P 36331454fb984af5a6d4f35efe0a2b00: Bootstrap starting.
I20240502 08:02:20.460438 7831 tablet_bootstrap.cc:492] T d4042d449ac74cb6bbeb7ba7f90131c4 P 16eac9aafbee42bd8a83544ec9b97e74: No bootstrap required, opened a new log
I20240502 08:02:20.460964 7831 ts_tablet_manager.cc:1367] T d4042d449ac74cb6bbeb7ba7f90131c4 P 16eac9aafbee42bd8a83544ec9b97e74: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.008s
I20240502 08:02:20.461300 7826 tablet_bootstrap.cc:654] T ac59612a1fa642f6b7dfd4d899bd6f9d P 36331454fb984af5a6d4f35efe0a2b00: Neither blocks nor log segments found. Creating new log.
I20240502 08:02:20.463580 7831 raft_consensus.cc:348] T d4042d449ac74cb6bbeb7ba7f90131c4 P 16eac9aafbee42bd8a83544ec9b97e74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } } peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } }
I20240502 08:02:20.464397 7831 raft_consensus.cc:374] T d4042d449ac74cb6bbeb7ba7f90131c4 P 16eac9aafbee42bd8a83544ec9b97e74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:02:20.464708 7831 raft_consensus.cc:729] T d4042d449ac74cb6bbeb7ba7f90131c4 P 16eac9aafbee42bd8a83544ec9b97e74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16eac9aafbee42bd8a83544ec9b97e74, State: Initialized, Role: FOLLOWER
I20240502 08:02:20.465567 7831 consensus_queue.cc:260] T d4042d449ac74cb6bbeb7ba7f90131c4 P 16eac9aafbee42bd8a83544ec9b97e74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } } peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } }
I20240502 08:02:20.468000 7831 ts_tablet_manager.cc:1398] T d4042d449ac74cb6bbeb7ba7f90131c4 P 16eac9aafbee42bd8a83544ec9b97e74: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20240502 08:02:20.468678 7827 tablet_bootstrap.cc:492] T 8e0f7109a1a94eb999a919ed98dea1ce P d5cb06cedbcb4941afcceab604f34ea4: No bootstrap required, opened a new log
I20240502 08:02:20.468958 7831 tablet_bootstrap.cc:492] T 8e0f7109a1a94eb999a919ed98dea1ce P 16eac9aafbee42bd8a83544ec9b97e74: Bootstrap starting.
I20240502 08:02:20.469478 7827 ts_tablet_manager.cc:1367] T 8e0f7109a1a94eb999a919ed98dea1ce P d5cb06cedbcb4941afcceab604f34ea4: Time spent bootstrapping tablet: real 0.034s user 0.016s sys 0.017s
I20240502 08:02:20.472569 7827 raft_consensus.cc:348] T 8e0f7109a1a94eb999a919ed98dea1ce P d5cb06cedbcb4941afcceab604f34ea4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } }
I20240502 08:02:20.473400 7827 raft_consensus.cc:374] T 8e0f7109a1a94eb999a919ed98dea1ce P d5cb06cedbcb4941afcceab604f34ea4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:02:20.473740 7827 raft_consensus.cc:729] T 8e0f7109a1a94eb999a919ed98dea1ce P d5cb06cedbcb4941afcceab604f34ea4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5cb06cedbcb4941afcceab604f34ea4, State: Initialized, Role: FOLLOWER
I20240502 08:02:20.474184 7826 tablet_bootstrap.cc:492] T ac59612a1fa642f6b7dfd4d899bd6f9d P 36331454fb984af5a6d4f35efe0a2b00: No bootstrap required, opened a new log
I20240502 08:02:20.474651 7826 ts_tablet_manager.cc:1367] T ac59612a1fa642f6b7dfd4d899bd6f9d P 36331454fb984af5a6d4f35efe0a2b00: Time spent bootstrapping tablet: real 0.022s user 0.015s sys 0.005s
I20240502 08:02:20.474684 7827 consensus_queue.cc:260] T 8e0f7109a1a94eb999a919ed98dea1ce P d5cb06cedbcb4941afcceab604f34ea4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } }
I20240502 08:02:20.474737 7831 tablet_bootstrap.cc:654] T 8e0f7109a1a94eb999a919ed98dea1ce P 16eac9aafbee42bd8a83544ec9b97e74: Neither blocks nor log segments found. Creating new log.
I20240502 08:02:20.479199 7827 ts_tablet_manager.cc:1398] T 8e0f7109a1a94eb999a919ed98dea1ce P d5cb06cedbcb4941afcceab604f34ea4: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20240502 08:02:20.480232 7827 tablet_bootstrap.cc:492] T b7961f100fcd49f581e2b1bfa1f1ec05 P d5cb06cedbcb4941afcceab604f34ea4: Bootstrap starting.
I20240502 08:02:20.480152 7826 raft_consensus.cc:348] T ac59612a1fa642f6b7dfd4d899bd6f9d P 36331454fb984af5a6d4f35efe0a2b00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } }
I20240502 08:02:20.481038 7826 raft_consensus.cc:374] T ac59612a1fa642f6b7dfd4d899bd6f9d P 36331454fb984af5a6d4f35efe0a2b00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:02:20.481364 7826 raft_consensus.cc:729] T ac59612a1fa642f6b7dfd4d899bd6f9d P 36331454fb984af5a6d4f35efe0a2b00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36331454fb984af5a6d4f35efe0a2b00, State: Initialized, Role: FOLLOWER
I20240502 08:02:20.482095 7826 consensus_queue.cc:260] T ac59612a1fa642f6b7dfd4d899bd6f9d P 36331454fb984af5a6d4f35efe0a2b00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } }
I20240502 08:02:20.486016 7831 tablet_bootstrap.cc:492] T 8e0f7109a1a94eb999a919ed98dea1ce P 16eac9aafbee42bd8a83544ec9b97e74: No bootstrap required, opened a new log
I20240502 08:02:20.486296 7827 tablet_bootstrap.cc:654] T b7961f100fcd49f581e2b1bfa1f1ec05 P d5cb06cedbcb4941afcceab604f34ea4: Neither blocks nor log segments found. Creating new log.
I20240502 08:02:20.486544 7831 ts_tablet_manager.cc:1367] T 8e0f7109a1a94eb999a919ed98dea1ce P 16eac9aafbee42bd8a83544ec9b97e74: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.003s
I20240502 08:02:20.488996 7831 raft_consensus.cc:348] T 8e0f7109a1a94eb999a919ed98dea1ce P 16eac9aafbee42bd8a83544ec9b97e74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } }
I20240502 08:02:20.489531 7831 raft_consensus.cc:374] T 8e0f7109a1a94eb999a919ed98dea1ce P 16eac9aafbee42bd8a83544ec9b97e74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:02:20.489768 7831 raft_consensus.cc:729] T 8e0f7109a1a94eb999a919ed98dea1ce P 16eac9aafbee42bd8a83544ec9b97e74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16eac9aafbee42bd8a83544ec9b97e74, State: Initialized, Role: FOLLOWER
I20240502 08:02:20.490345 7831 consensus_queue.cc:260] T 8e0f7109a1a94eb999a919ed98dea1ce P 16eac9aafbee42bd8a83544ec9b97e74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } }
I20240502 08:02:20.491779 7826 ts_tablet_manager.cc:1398] T ac59612a1fa642f6b7dfd4d899bd6f9d P 36331454fb984af5a6d4f35efe0a2b00: Time spent starting tablet: real 0.016s user 0.005s sys 0.000s
I20240502 08:02:20.492338 7831 ts_tablet_manager.cc:1398] T 8e0f7109a1a94eb999a919ed98dea1ce P 16eac9aafbee42bd8a83544ec9b97e74: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240502 08:02:20.493402 7831 tablet_bootstrap.cc:492] T f8a4589c0a9b417188a2d884011b0240 P 16eac9aafbee42bd8a83544ec9b97e74: Bootstrap starting.
I20240502 08:02:20.496930 7826 tablet_bootstrap.cc:492] T af8197b1de49493a966b8ba1146ada68 P 36331454fb984af5a6d4f35efe0a2b00: Bootstrap starting.
I20240502 08:02:20.497537 7827 tablet_bootstrap.cc:492] T b7961f100fcd49f581e2b1bfa1f1ec05 P d5cb06cedbcb4941afcceab604f34ea4: No bootstrap required, opened a new log
I20240502 08:02:20.498044 7834 raft_consensus.cc:482] T cb9b5608afa2415e905426d7400abd6c P 16eac9aafbee42bd8a83544ec9b97e74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:02:20.498796 7831 tablet_bootstrap.cc:654] T f8a4589c0a9b417188a2d884011b0240 P 16eac9aafbee42bd8a83544ec9b97e74: Neither blocks nor log segments found. Creating new log.
I20240502 08:02:20.498553 7834 raft_consensus.cc:504] T cb9b5608afa2415e905426d7400abd6c P 16eac9aafbee42bd8a83544ec9b97e74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } }
I20240502 08:02:20.502774 7826 tablet_bootstrap.cc:654] T af8197b1de49493a966b8ba1146ada68 P 36331454fb984af5a6d4f35efe0a2b00: Neither blocks nor log segments found. Creating new log.
I20240502 08:02:20.511839 7834 leader_election.cc:288] T cb9b5608afa2415e905426d7400abd6c P 16eac9aafbee42bd8a83544ec9b97e74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d5cb06cedbcb4941afcceab604f34ea4 (127.6.204.193:42525), 36331454fb984af5a6d4f35efe0a2b00 (127.6.204.195:40297)
I20240502 08:02:20.512264 7826 tablet_bootstrap.cc:492] T af8197b1de49493a966b8ba1146ada68 P 36331454fb984af5a6d4f35efe0a2b00: No bootstrap required, opened a new log
I20240502 08:02:20.512936 7826 ts_tablet_manager.cc:1367] T af8197b1de49493a966b8ba1146ada68 P 36331454fb984af5a6d4f35efe0a2b00: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.000s
I20240502 08:02:20.516247 7826 raft_consensus.cc:348] T af8197b1de49493a966b8ba1146ada68 P 36331454fb984af5a6d4f35efe0a2b00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } }
I20240502 08:02:20.498028 7827 ts_tablet_manager.cc:1367] T b7961f100fcd49f581e2b1bfa1f1ec05 P d5cb06cedbcb4941afcceab604f34ea4: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.006s
I20240502 08:02:20.520659 7827 raft_consensus.cc:348] T b7961f100fcd49f581e2b1bfa1f1ec05 P d5cb06cedbcb4941afcceab604f34ea4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } }
I20240502 08:02:20.521548 7827 raft_consensus.cc:374] T b7961f100fcd49f581e2b1bfa1f1ec05 P d5cb06cedbcb4941afcceab604f34ea4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:02:20.517248 7826 raft_consensus.cc:374] T af8197b1de49493a966b8ba1146ada68 P 36331454fb984af5a6d4f35efe0a2b00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:02:20.522300 7826 raft_consensus.cc:729] T af8197b1de49493a966b8ba1146ada68 P 36331454fb984af5a6d4f35efe0a2b00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36331454fb984af5a6d4f35efe0a2b00, State: Initialized, Role: FOLLOWER
I20240502 08:02:20.523201 7826 consensus_queue.cc:260] T af8197b1de49493a966b8ba1146ada68 P 36331454fb984af5a6d4f35efe0a2b00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } }
I20240502 08:02:20.524536 7830 raft_consensus.cc:482] T af8197b1de49493a966b8ba1146ada68 P d5cb06cedbcb4941afcceab604f34ea4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:02:20.525184 7830 raft_consensus.cc:504] T af8197b1de49493a966b8ba1146ada68 P d5cb06cedbcb4941afcceab604f34ea4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } }
I20240502 08:02:20.528240 7826 ts_tablet_manager.cc:1398] T af8197b1de49493a966b8ba1146ada68 P 36331454fb984af5a6d4f35efe0a2b00: Time spent starting tablet: real 0.015s user 0.008s sys 0.002s
I20240502 08:02:20.529418 7826 tablet_bootstrap.cc:492] T 8e0f7109a1a94eb999a919ed98dea1ce P 36331454fb984af5a6d4f35efe0a2b00: Bootstrap starting.
I20240502 08:02:20.521937 7827 raft_consensus.cc:729] T b7961f100fcd49f581e2b1bfa1f1ec05 P d5cb06cedbcb4941afcceab604f34ea4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5cb06cedbcb4941afcceab604f34ea4, State: Initialized, Role: FOLLOWER
I20240502 08:02:20.530906 7632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb9b5608afa2415e905426d7400abd6c" candidate_uuid: "16eac9aafbee42bd8a83544ec9b97e74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5cb06cedbcb4941afcceab604f34ea4" is_pre_election: true
I20240502 08:02:20.530684 7827 consensus_queue.cc:260] T b7961f100fcd49f581e2b1bfa1f1ec05 P d5cb06cedbcb4941afcceab604f34ea4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } }
W20240502 08:02:20.532701 7670 leader_election.cc:341] T cb9b5608afa2415e905426d7400abd6c P 16eac9aafbee42bd8a83544ec9b97e74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d5cb06cedbcb4941afcceab604f34ea4 (127.6.204.193:42525): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:02:20.540990 7830 leader_election.cc:288] T af8197b1de49493a966b8ba1146ada68 P d5cb06cedbcb4941afcceab604f34ea4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 16eac9aafbee42bd8a83544ec9b97e74 (127.6.204.194:38399), 36331454fb984af5a6d4f35efe0a2b00 (127.6.204.195:40297)
I20240502 08:02:20.550565 7827 ts_tablet_manager.cc:1398] T b7961f100fcd49f581e2b1bfa1f1ec05 P d5cb06cedbcb4941afcceab604f34ea4: Time spent starting tablet: real 0.033s user 0.004s sys 0.003s
I20240502 08:02:20.551782 7827 tablet_bootstrap.cc:492] T 8f45c7e5e5cf4b4a84127573dd446bde P d5cb06cedbcb4941afcceab604f34ea4: Bootstrap starting.
I20240502 08:02:20.551826 7826 tablet_bootstrap.cc:654] T 8e0f7109a1a94eb999a919ed98dea1ce P 36331454fb984af5a6d4f35efe0a2b00: Neither blocks nor log segments found. Creating new log.
I20240502 08:02:20.555801 7831 tablet_bootstrap.cc:492] T f8a4589c0a9b417188a2d884011b0240 P 16eac9aafbee42bd8a83544ec9b97e74: No bootstrap required, opened a new log
I20240502 08:02:20.558549 7831 ts_tablet_manager.cc:1367] T f8a4589c0a9b417188a2d884011b0240 P 16eac9aafbee42bd8a83544ec9b97e74: Time spent bootstrapping tablet: real 0.065s user 0.027s sys 0.027s
I20240502 08:02:20.559278 7781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb9b5608afa2415e905426d7400abd6c" candidate_uuid: "16eac9aafbee42bd8a83544ec9b97e74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36331454fb984af5a6d4f35efe0a2b00" is_pre_election: true
I20240502 08:02:20.561434 7831 raft_consensus.cc:348] T f8a4589c0a9b417188a2d884011b0240 P 16eac9aafbee42bd8a83544ec9b97e74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } }
I20240502 08:02:20.561993 7831 raft_consensus.cc:374] T f8a4589c0a9b417188a2d884011b0240 P 16eac9aafbee42bd8a83544ec9b97e74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:02:20.560024 7781 raft_consensus.cc:2454] T cb9b5608afa2415e905426d7400abd6c P 36331454fb984af5a6d4f35efe0a2b00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16eac9aafbee42bd8a83544ec9b97e74 in term 0.
I20240502 08:02:20.562304 7831 raft_consensus.cc:729] T f8a4589c0a9b417188a2d884011b0240 P 16eac9aafbee42bd8a83544ec9b97e74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16eac9aafbee42bd8a83544ec9b97e74, State: Initialized, Role: FOLLOWER
I20240502 08:02:20.563032 7831 consensus_queue.cc:260] T f8a4589c0a9b417188a2d884011b0240 P 16eac9aafbee42bd8a83544ec9b97e74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } }
I20240502 08:02:20.563508 7671 leader_election.cc:302] T cb9b5608afa2415e905426d7400abd6c P 16eac9aafbee42bd8a83544ec9b97e74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 16eac9aafbee42bd8a83544ec9b97e74, 36331454fb984af5a6d4f35efe0a2b00; no voters: d5cb06cedbcb4941afcceab604f34ea4
I20240502 08:02:20.565060 7834 raft_consensus.cc:2781] T cb9b5608afa2415e905426d7400abd6c P 16eac9aafbee42bd8a83544ec9b97e74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:02:20.565428 7834 raft_consensus.cc:482] T cb9b5608afa2415e905426d7400abd6c P 16eac9aafbee42bd8a83544ec9b97e74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:02:20.565891 7834 raft_consensus.cc:3037] T cb9b5608afa2415e905426d7400abd6c P 16eac9aafbee42bd8a83544ec9b97e74 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:02:20.570619 7831 ts_tablet_manager.cc:1398] T f8a4589c0a9b417188a2d884011b0240 P 16eac9aafbee42bd8a83544ec9b97e74: Time spent starting tablet: real 0.012s user 0.006s sys 0.001s
I20240502 08:02:20.573836 7827 tablet_bootstrap.cc:654] T 8f45c7e5e5cf4b4a84127573dd446bde P d5cb06cedbcb4941afcceab604f34ea4: Neither blocks nor log segments found. Creating new log.
I20240502 08:02:20.577533 7831 tablet_bootstrap.cc:492] T b7961f100fcd49f581e2b1bfa1f1ec05 P 16eac9aafbee42bd8a83544ec9b97e74: Bootstrap starting.
I20240502 08:02:20.578162 7707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af8197b1de49493a966b8ba1146ada68" candidate_uuid: "d5cb06cedbcb4941afcceab604f34ea4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16eac9aafbee42bd8a83544ec9b97e74" is_pre_election: true
W20240502 08:02:20.581305 7597 leader_election.cc:341] T af8197b1de49493a966b8ba1146ada68 P d5cb06cedbcb4941afcceab604f34ea4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 16eac9aafbee42bd8a83544ec9b97e74 (127.6.204.194:38399): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:02:20.582881 7834 raft_consensus.cc:504] T cb9b5608afa2415e905426d7400abd6c P 16eac9aafbee42bd8a83544ec9b97e74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } }
I20240502 08:02:20.586675 7781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af8197b1de49493a966b8ba1146ada68" candidate_uuid: "d5cb06cedbcb4941afcceab604f34ea4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36331454fb984af5a6d4f35efe0a2b00" is_pre_election: true
I20240502 08:02:20.587385 7781 raft_consensus.cc:2454] T af8197b1de49493a966b8ba1146ada68 P 36331454fb984af5a6d4f35efe0a2b00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5cb06cedbcb4941afcceab604f34ea4 in term 0.
I20240502 08:02:20.588594 7831 tablet_bootstrap.cc:654] T b7961f100fcd49f581e2b1bfa1f1ec05 P 16eac9aafbee42bd8a83544ec9b97e74: Neither blocks nor log segments found. Creating new log.
I20240502 08:02:20.588845 7596 leader_election.cc:302] T af8197b1de49493a966b8ba1146ada68 P d5cb06cedbcb4941afcceab604f34ea4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 36331454fb984af5a6d4f35efe0a2b00, d5cb06cedbcb4941afcceab604f34ea4; no voters: 16eac9aafbee42bd8a83544ec9b97e74
I20240502 08:02:20.590034 7830 raft_consensus.cc:2781] T af8197b1de49493a966b8ba1146ada68 P d5cb06cedbcb4941afcceab604f34ea4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:02:20.590418 7830 raft_consensus.cc:482] T af8197b1de49493a966b8ba1146ada68 P d5cb06cedbcb4941afcceab604f34ea4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:02:20.590796 7827 tablet_bootstrap.cc:492] T 8f45c7e5e5cf4b4a84127573dd446bde P d5cb06cedbcb4941afcceab604f34ea4: No bootstrap required, opened a new log
I20240502 08:02:20.590755 7830 raft_consensus.cc:3037] T af8197b1de49493a966b8ba1146ada68 P d5cb06cedbcb4941afcceab604f34ea4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:02:20.591336 7826 tablet_bootstrap.cc:492] T 8e0f7109a1a94eb999a919ed98dea1ce P 36331454fb984af5a6d4f35efe0a2b00: No bootstrap required, opened a new log
I20240502 08:02:20.591820 7826 ts_tablet_manager.cc:1367] T 8e0f7109a1a94eb999a919ed98dea1ce P 36331454fb984af5a6d4f35efe0a2b00: Time spent bootstrapping tablet: real 0.063s user 0.012s sys 0.010s
I20240502 08:02:20.592767 7834 leader_election.cc:288] T cb9b5608afa2415e905426d7400abd6c P 16eac9aafbee42bd8a83544ec9b97e74 [CANDIDATE]: Term 1 election: Requested vote from peers d5cb06cedbcb4941afcceab604f34ea4 (127.6.204.193:42525), 36331454fb984af5a6d4f35efe0a2b00 (127.6.204.195:40297)
I20240502 08:02:20.593868 7781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb9b5608afa2415e905426d7400abd6c" candidate_uuid: "16eac9aafbee42bd8a83544ec9b97e74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36331454fb984af5a6d4f35efe0a2b00"
I20240502 08:02:20.594496 7781 raft_consensus.cc:3037] T cb9b5608afa2415e905426d7400abd6c P 36331454fb984af5a6d4f35efe0a2b00 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:02:20.596706 7826 raft_consensus.cc:348] T 8e0f7109a1a94eb999a919ed98dea1ce P 36331454fb984af5a6d4f35efe0a2b00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } }
I20240502 08:02:20.597450 7826 raft_consensus.cc:374] T 8e0f7109a1a94eb999a919ed98dea1ce P 36331454fb984af5a6d4f35efe0a2b00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:02:20.597726 7826 raft_consensus.cc:729] T 8e0f7109a1a94eb999a919ed98dea1ce P 36331454fb984af5a6d4f35efe0a2b00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36331454fb984af5a6d4f35efe0a2b00, State: Initialized, Role: FOLLOWER
I20240502 08:02:20.598505 7632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb9b5608afa2415e905426d7400abd6c" candidate_uuid: "16eac9aafbee42bd8a83544ec9b97e74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5cb06cedbcb4941afcceab604f34ea4"
I20240502 08:02:20.598451 7826 consensus_queue.cc:260] T 8e0f7109a1a94eb999a919ed98dea1ce P 36331454fb984af5a6d4f35efe0a2b00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } }
I20240502 08:02:20.599345 7781 raft_consensus.cc:2454] T cb9b5608afa2415e905426d7400abd6c P 36331454fb984af5a6d4f35efe0a2b00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16eac9aafbee42bd8a83544ec9b97e74 in term 1.
W20240502 08:02:20.599874 7670 leader_election.cc:341] T cb9b5608afa2415e905426d7400abd6c P 16eac9aafbee42bd8a83544ec9b97e74 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d5cb06cedbcb4941afcceab604f34ea4 (127.6.204.193:42525): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:02:20.600529 7671 leader_election.cc:302] T cb9b5608afa2415e905426d7400abd6c P 16eac9aafbee42bd8a83544ec9b97e74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 16eac9aafbee42bd8a83544ec9b97e74, 36331454fb984af5a6d4f35efe0a2b00; no voters: d5cb06cedbcb4941afcceab604f34ea4
I20240502 08:02:20.601732 7834 raft_consensus.cc:2781] T cb9b5608afa2415e905426d7400abd6c P 16eac9aafbee42bd8a83544ec9b97e74 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:02:20.602149 7831 tablet_bootstrap.cc:492] T b7961f100fcd49f581e2b1bfa1f1ec05 P 16eac9aafbee42bd8a83544ec9b97e74: No bootstrap required, opened a new log
I20240502 08:02:20.602664 7831 ts_tablet_manager.cc:1367] T b7961f100fcd49f581e2b1bfa1f1ec05 P 16eac9aafbee42bd8a83544ec9b97e74: Time spent bootstrapping tablet: real 0.028s user 0.010s sys 0.003s
I20240502 08:02:20.602941 7826 ts_tablet_manager.cc:1398] T 8e0f7109a1a94eb999a919ed98dea1ce P 36331454fb984af5a6d4f35efe0a2b00: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20240502 08:02:20.603915 7826 tablet_bootstrap.cc:492] T b7961f100fcd49f581e2b1bfa1f1ec05 P 36331454fb984af5a6d4f35efe0a2b00: Bootstrap starting.
I20240502 08:02:20.604429 7827 ts_tablet_manager.cc:1367] T 8f45c7e5e5cf4b4a84127573dd446bde P d5cb06cedbcb4941afcceab604f34ea4: Time spent bootstrapping tablet: real 0.053s user 0.014s sys 0.004s
I20240502 08:02:20.606223 7831 raft_consensus.cc:348] T b7961f100fcd49f581e2b1bfa1f1ec05 P 16eac9aafbee42bd8a83544ec9b97e74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } }
I20240502 08:02:20.607304 7831 raft_consensus.cc:374] T b7961f100fcd49f581e2b1bfa1f1ec05 P 16eac9aafbee42bd8a83544ec9b97e74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:02:20.607688 7831 raft_consensus.cc:729] T b7961f100fcd49f581e2b1bfa1f1ec05 P 16eac9aafbee42bd8a83544ec9b97e74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16eac9aafbee42bd8a83544ec9b97e74, State: Initialized, Role: FOLLOWER
I20240502 08:02:20.605620 7830 raft_consensus.cc:504] T af8197b1de49493a966b8ba1146ada68 P d5cb06cedbcb4941afcceab604f34ea4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } }
I20240502 08:02:20.608606 7831 consensus_queue.cc:260] T b7961f100fcd49f581e2b1bfa1f1ec05 P 16eac9aafbee42bd8a83544ec9b97e74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } }
I20240502 08:02:20.609642 7826 tablet_bootstrap.cc:654] T b7961f100fcd49f581e2b1bfa1f1ec05 P 36331454fb984af5a6d4f35efe0a2b00: Neither blocks nor log segments found. Creating new log.
I20240502 08:02:20.610457 7707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af8197b1de49493a966b8ba1146ada68" candidate_uuid: "d5cb06cedbcb4941afcceab604f34ea4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16eac9aafbee42bd8a83544ec9b97e74"
W20240502 08:02:20.612301 7597 leader_election.cc:341] T af8197b1de49493a966b8ba1146ada68 P d5cb06cedbcb4941afcceab604f34ea4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 16eac9aafbee42bd8a83544ec9b97e74 (127.6.204.194:38399): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:02:20.615303 7781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af8197b1de49493a966b8ba1146ada68" candidate_uuid: "d5cb06cedbcb4941afcceab604f34ea4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36331454fb984af5a6d4f35efe0a2b00"
I20240502 08:02:20.616142 7781 raft_consensus.cc:3037] T af8197b1de49493a966b8ba1146ada68 P 36331454fb984af5a6d4f35efe0a2b00 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:02:20.619313 7848 raft_consensus.cc:482] T d4042d449ac74cb6bbeb7ba7f90131c4 P d5cb06cedbcb4941afcceab604f34ea4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:02:20.620100 7831 ts_tablet_manager.cc:1398] T b7961f100fcd49f581e2b1bfa1f1ec05 P 16eac9aafbee42bd8a83544ec9b97e74: Time spent starting tablet: real 0.017s user 0.009s sys 0.007s
I20240502 08:02:20.619799 7848 raft_consensus.cc:504] T d4042d449ac74cb6bbeb7ba7f90131c4 P d5cb06cedbcb4941afcceab604f34ea4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } } peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } }
I20240502 08:02:20.621642 7848 leader_election.cc:288] T d4042d449ac74cb6bbeb7ba7f90131c4 P d5cb06cedbcb4941afcceab604f34ea4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 36331454fb984af5a6d4f35efe0a2b00 (127.6.204.195:40297), 16eac9aafbee42bd8a83544ec9b97e74 (127.6.204.194:38399)
I20240502 08:02:20.622876 7780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4042d449ac74cb6bbeb7ba7f90131c4" candidate_uuid: "d5cb06cedbcb4941afcceab604f34ea4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36331454fb984af5a6d4f35efe0a2b00" is_pre_election: true
I20240502 08:02:20.623114 7707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4042d449ac74cb6bbeb7ba7f90131c4" candidate_uuid: "d5cb06cedbcb4941afcceab604f34ea4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16eac9aafbee42bd8a83544ec9b97e74" is_pre_election: true
I20240502 08:02:20.623567 7780 raft_consensus.cc:2454] T d4042d449ac74cb6bbeb7ba7f90131c4 P 36331454fb984af5a6d4f35efe0a2b00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5cb06cedbcb4941afcceab604f34ea4 in term 0.
I20240502 08:02:20.623706 7707 raft_consensus.cc:2454] T d4042d449ac74cb6bbeb7ba7f90131c4 P 16eac9aafbee42bd8a83544ec9b97e74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5cb06cedbcb4941afcceab604f34ea4 in term 0.
I20240502 08:02:20.624761 7597 leader_election.cc:302] T d4042d449ac74cb6bbeb7ba7f90131c4 P d5cb06cedbcb4941afcceab604f34ea4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16eac9aafbee42bd8a83544ec9b97e74, d5cb06cedbcb4941afcceab604f34ea4; no voters:
I20240502 08:02:20.626224 7848 raft_consensus.cc:2781] T d4042d449ac74cb6bbeb7ba7f90131c4 P d5cb06cedbcb4941afcceab604f34ea4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:02:20.626614 7848 raft_consensus.cc:482] T d4042d449ac74cb6bbeb7ba7f90131c4 P d5cb06cedbcb4941afcceab604f34ea4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:02:20.627090 7848 raft_consensus.cc:3037] T d4042d449ac74cb6bbeb7ba7f90131c4 P d5cb06cedbcb4941afcceab604f34ea4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:02:20.626745 7827 raft_consensus.cc:348] T 8f45c7e5e5cf4b4a84127573dd446bde P d5cb06cedbcb4941afcceab604f34ea4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } }
I20240502 08:02:20.630308 7827 raft_consensus.cc:374] T 8f45c7e5e5cf4b4a84127573dd446bde P d5cb06cedbcb4941afcceab604f34ea4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:02:20.630650 7827 raft_consensus.cc:729] T 8f45c7e5e5cf4b4a84127573dd446bde P d5cb06cedbcb4941afcceab604f34ea4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5cb06cedbcb4941afcceab604f34ea4, State: Initialized, Role: FOLLOWER
I20240502 08:02:20.629796 7781 raft_consensus.cc:2454] T af8197b1de49493a966b8ba1146ada68 P 36331454fb984af5a6d4f35efe0a2b00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5cb06cedbcb4941afcceab604f34ea4 in term 1.
I20240502 08:02:20.632107 7596 leader_election.cc:302] T af8197b1de49493a966b8ba1146ada68 P d5cb06cedbcb4941afcceab604f34ea4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 36331454fb984af5a6d4f35efe0a2b00, d5cb06cedbcb4941afcceab604f34ea4; no voters: 16eac9aafbee42bd8a83544ec9b97e74
I20240502 08:02:20.632669 7826 tablet_bootstrap.cc:492] T b7961f100fcd49f581e2b1bfa1f1ec05 P 36331454fb984af5a6d4f35efe0a2b00: No bootstrap required, opened a new log
I20240502 08:02:20.633178 7826 ts_tablet_manager.cc:1367] T b7961f100fcd49f581e2b1bfa1f1ec05 P 36331454fb984af5a6d4f35efe0a2b00: Time spent bootstrapping tablet: real 0.029s user 0.010s sys 0.016s
I20240502 08:02:20.627112 7830 leader_election.cc:288] T af8197b1de49493a966b8ba1146ada68 P d5cb06cedbcb4941afcceab604f34ea4 [CANDIDATE]: Term 1 election: Requested vote from peers 16eac9aafbee42bd8a83544ec9b97e74 (127.6.204.194:38399), 36331454fb984af5a6d4f35efe0a2b00 (127.6.204.195:40297)
I20240502 08:02:20.634573 7830 raft_consensus.cc:2781] T af8197b1de49493a966b8ba1146ada68 P d5cb06cedbcb4941afcceab604f34ea4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:02:20.634922 7831 tablet_bootstrap.cc:492] T af8197b1de49493a966b8ba1146ada68 P 16eac9aafbee42bd8a83544ec9b97e74: Bootstrap starting.
I20240502 08:02:20.635001 7848 raft_consensus.cc:504] T d4042d449ac74cb6bbeb7ba7f90131c4 P d5cb06cedbcb4941afcceab604f34ea4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } } peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } }
I20240502 08:02:20.635020 7830 raft_consensus.cc:686] T af8197b1de49493a966b8ba1146ada68 P d5cb06cedbcb4941afcceab604f34ea4 [term 1 LEADER]: Becoming Leader. State: Replica: d5cb06cedbcb4941afcceab604f34ea4, State: Running, Role: LEADER
I20240502 08:02:20.640904 7830 consensus_queue.cc:237] T af8197b1de49493a966b8ba1146ada68 P d5cb06cedbcb4941afcceab604f34ea4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } }
I20240502 08:02:20.635910 7826 raft_consensus.cc:348] T b7961f100fcd49f581e2b1bfa1f1ec05 P 36331454fb984af5a6d4f35efe0a2b00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } }
I20240502 08:02:20.642722 7707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4042d449ac74cb6bbeb7ba7f90131c4" candidate_uuid: "d5cb06cedbcb4941afcceab604f34ea4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16eac9aafbee42bd8a83544ec9b97e74"
I20240502 08:02:20.643369 7707 raft_consensus.cc:3037] T d4042d449ac74cb6bbeb7ba7f90131c4 P 16eac9aafbee42bd8a83544ec9b97e74 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:02:20.649252 7831 tablet_bootstrap.cc:654] T af8197b1de49493a966b8ba1146ada68 P 16eac9aafbee42bd8a83544ec9b97e74: Neither blocks nor log segments found. Creating new log.
I20240502 08:02:20.642819 7826 raft_consensus.cc:374] T b7961f100fcd49f581e2b1bfa1f1ec05 P 36331454fb984af5a6d4f35efe0a2b00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:02:20.650681 7848 leader_election.cc:288] T d4042d449ac74cb6bbeb7ba7f90131c4 P d5cb06cedbcb4941afcceab604f34ea4 [CANDIDATE]: Term 1 election: Requested vote from peers 36331454fb984af5a6d4f35efe0a2b00 (127.6.204.195:40297), 16eac9aafbee42bd8a83544ec9b97e74 (127.6.204.194:38399)
I20240502 08:02:20.636763 7827 consensus_queue.cc:260] T 8f45c7e5e5cf4b4a84127573dd446bde P d5cb06cedbcb4941afcceab604f34ea4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } }
I20240502 08:02:20.650241 7707 raft_consensus.cc:2454] T d4042d449ac74cb6bbeb7ba7f90131c4 P 16eac9aafbee42bd8a83544ec9b97e74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5cb06cedbcb4941afcceab604f34ea4 in term 1.
I20240502 08:02:20.650969 7528 catalog_manager.cc:5484] T af8197b1de49493a966b8ba1146ada68 P d5cb06cedbcb4941afcceab604f34ea4 reported cstate change: term changed from 0 to 1, leader changed from <none> to d5cb06cedbcb4941afcceab604f34ea4 (127.6.204.193). New cstate: current_term: 1 leader_uuid: "d5cb06cedbcb4941afcceab604f34ea4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } health_report { overall_health: UNKNOWN } } }
I20240502 08:02:20.651801 7781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4042d449ac74cb6bbeb7ba7f90131c4" candidate_uuid: "d5cb06cedbcb4941afcceab604f34ea4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36331454fb984af5a6d4f35efe0a2b00"
I20240502 08:02:20.650872 7826 raft_consensus.cc:729] T b7961f100fcd49f581e2b1bfa1f1ec05 P 36331454fb984af5a6d4f35efe0a2b00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36331454fb984af5a6d4f35efe0a2b00, State: Initialized, Role: FOLLOWER
I20240502 08:02:20.652484 7781 raft_consensus.cc:3037] T d4042d449ac74cb6bbeb7ba7f90131c4 P 36331454fb984af5a6d4f35efe0a2b00 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:02:20.653252 7827 ts_tablet_manager.cc:1398] T 8f45c7e5e5cf4b4a84127573dd446bde P d5cb06cedbcb4941afcceab604f34ea4: Time spent starting tablet: real 0.047s user 0.006s sys 0.001s
I20240502 08:02:20.653195 7826 consensus_queue.cc:260] T b7961f100fcd49f581e2b1bfa1f1ec05 P 36331454fb984af5a6d4f35efe0a2b00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } }
I20240502 08:02:20.654539 7827 tablet_bootstrap.cc:492] T cb9b5608afa2415e905426d7400abd6c P d5cb06cedbcb4941afcceab604f34ea4: Bootstrap starting.
I20240502 08:02:20.654461 7597 leader_election.cc:302] T d4042d449ac74cb6bbeb7ba7f90131c4 P d5cb06cedbcb4941afcceab604f34ea4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16eac9aafbee42bd8a83544ec9b97e74, d5cb06cedbcb4941afcceab604f34ea4; no voters:
I20240502 08:02:20.655889 7826 ts_tablet_manager.cc:1398] T b7961f100fcd49f581e2b1bfa1f1ec05 P 36331454fb984af5a6d4f35efe0a2b00: Time spent starting tablet: real 0.022s user 0.004s sys 0.004s
I20240502 08:02:20.658318 7854 raft_consensus.cc:482] T ac59612a1fa642f6b7dfd4d899bd6f9d P 16eac9aafbee42bd8a83544ec9b97e74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:02:20.658813 7854 raft_consensus.cc:504] T ac59612a1fa642f6b7dfd4d899bd6f9d P 16eac9aafbee42bd8a83544ec9b97e74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } }
I20240502 08:02:20.659302 7781 raft_consensus.cc:2454] T d4042d449ac74cb6bbeb7ba7f90131c4 P 36331454fb984af5a6d4f35efe0a2b00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5cb06cedbcb4941afcceab604f34ea4 in term 1.
I20240502 08:02:20.638693 7834 raft_consensus.cc:686] T cb9b5608afa2415e905426d7400abd6c P 16eac9aafbee42bd8a83544ec9b97e74 [term 1 LEADER]: Becoming Leader. State: Replica: 16eac9aafbee42bd8a83544ec9b97e74, State: Running, Role: LEADER
I20240502 08:02:20.661276 7834 consensus_queue.cc:237] T cb9b5608afa2415e905426d7400abd6c P 16eac9aafbee42bd8a83544ec9b97e74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } }
I20240502 08:02:20.663547 7848 raft_consensus.cc:2781] T d4042d449ac74cb6bbeb7ba7f90131c4 P d5cb06cedbcb4941afcceab604f34ea4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:02:20.664053 7848 raft_consensus.cc:686] T d4042d449ac74cb6bbeb7ba7f90131c4 P d5cb06cedbcb4941afcceab604f34ea4 [term 1 LEADER]: Becoming Leader. State: Replica: d5cb06cedbcb4941afcceab604f34ea4, State: Running, Role: LEADER
I20240502 08:02:20.666502 7632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac59612a1fa642f6b7dfd4d899bd6f9d" candidate_uuid: "16eac9aafbee42bd8a83544ec9b97e74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5cb06cedbcb4941afcceab604f34ea4" is_pre_election: true
I20240502 08:02:20.668522 7827 tablet_bootstrap.cc:654] T cb9b5608afa2415e905426d7400abd6c P d5cb06cedbcb4941afcceab604f34ea4: Neither blocks nor log segments found. Creating new log.
W20240502 08:02:20.670354 7670 leader_election.cc:341] T ac59612a1fa642f6b7dfd4d899bd6f9d P 16eac9aafbee42bd8a83544ec9b97e74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d5cb06cedbcb4941afcceab604f34ea4 (127.6.204.193:42525): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:02:20.668493 7854 leader_election.cc:288] T ac59612a1fa642f6b7dfd4d899bd6f9d P 16eac9aafbee42bd8a83544ec9b97e74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d5cb06cedbcb4941afcceab604f34ea4 (127.6.204.193:42525), 36331454fb984af5a6d4f35efe0a2b00 (127.6.204.195:40297)
I20240502 08:02:20.669016 7848 consensus_queue.cc:237] T d4042d449ac74cb6bbeb7ba7f90131c4 P d5cb06cedbcb4941afcceab604f34ea4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } } peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } }
I20240502 08:02:20.674857 7781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac59612a1fa642f6b7dfd4d899bd6f9d" candidate_uuid: "16eac9aafbee42bd8a83544ec9b97e74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36331454fb984af5a6d4f35efe0a2b00" is_pre_election: true
I20240502 08:02:20.675655 7781 raft_consensus.cc:2454] T ac59612a1fa642f6b7dfd4d899bd6f9d P 36331454fb984af5a6d4f35efe0a2b00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16eac9aafbee42bd8a83544ec9b97e74 in term 0.
I20240502 08:02:20.675873 7826 tablet_bootstrap.cc:492] T f8a4589c0a9b417188a2d884011b0240 P 36331454fb984af5a6d4f35efe0a2b00: Bootstrap starting.
I20240502 08:02:20.678153 7671 leader_election.cc:302] T ac59612a1fa642f6b7dfd4d899bd6f9d P 16eac9aafbee42bd8a83544ec9b97e74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 16eac9aafbee42bd8a83544ec9b97e74, 36331454fb984af5a6d4f35efe0a2b00; no voters: d5cb06cedbcb4941afcceab604f34ea4
I20240502 08:02:20.679114 7854 raft_consensus.cc:2781] T ac59612a1fa642f6b7dfd4d899bd6f9d P 16eac9aafbee42bd8a83544ec9b97e74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:02:20.679634 7854 raft_consensus.cc:482] T ac59612a1fa642f6b7dfd4d899bd6f9d P 16eac9aafbee42bd8a83544ec9b97e74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:02:20.680305 7854 raft_consensus.cc:3037] T ac59612a1fa642f6b7dfd4d899bd6f9d P 16eac9aafbee42bd8a83544ec9b97e74 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:02:20.683544 7831 tablet_bootstrap.cc:492] T af8197b1de49493a966b8ba1146ada68 P 16eac9aafbee42bd8a83544ec9b97e74: No bootstrap required, opened a new log
I20240502 08:02:20.686718 7831 ts_tablet_manager.cc:1367] T af8197b1de49493a966b8ba1146ada68 P 16eac9aafbee42bd8a83544ec9b97e74: Time spent bootstrapping tablet: real 0.053s user 0.019s sys 0.010s
I20240502 08:02:20.685194 7827 tablet_bootstrap.cc:492] T cb9b5608afa2415e905426d7400abd6c P d5cb06cedbcb4941afcceab604f34ea4: No bootstrap required, opened a new log
I20240502 08:02:20.691316 7831 raft_consensus.cc:348] T af8197b1de49493a966b8ba1146ada68 P 16eac9aafbee42bd8a83544ec9b97e74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } }
I20240502 08:02:20.692335 7831 raft_consensus.cc:374] T af8197b1de49493a966b8ba1146ada68 P 16eac9aafbee42bd8a83544ec9b97e74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:02:20.692719 7831 raft_consensus.cc:729] T af8197b1de49493a966b8ba1146ada68 P 16eac9aafbee42bd8a83544ec9b97e74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16eac9aafbee42bd8a83544ec9b97e74, State: Initialized, Role: FOLLOWER
I20240502 08:02:20.693775 7854 raft_consensus.cc:504] T ac59612a1fa642f6b7dfd4d899bd6f9d P 16eac9aafbee42bd8a83544ec9b97e74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } }
I20240502 08:02:20.694396 7827 ts_tablet_manager.cc:1367] T cb9b5608afa2415e905426d7400abd6c P d5cb06cedbcb4941afcceab604f34ea4: Time spent bootstrapping tablet: real 0.040s user 0.011s sys 0.003s
I20240502 08:02:20.696779 7632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac59612a1fa642f6b7dfd4d899bd6f9d" candidate_uuid: "16eac9aafbee42bd8a83544ec9b97e74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5cb06cedbcb4941afcceab604f34ea4"
I20240502 08:02:20.700186 7781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac59612a1fa642f6b7dfd4d899bd6f9d" candidate_uuid: "16eac9aafbee42bd8a83544ec9b97e74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36331454fb984af5a6d4f35efe0a2b00"
I20240502 08:02:20.704063 7781 raft_consensus.cc:3037] T ac59612a1fa642f6b7dfd4d899bd6f9d P 36331454fb984af5a6d4f35efe0a2b00 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:02:20.700264 7854 leader_election.cc:288] T ac59612a1fa642f6b7dfd4d899bd6f9d P 16eac9aafbee42bd8a83544ec9b97e74 [CANDIDATE]: Term 1 election: Requested vote from peers d5cb06cedbcb4941afcceab604f34ea4 (127.6.204.193:42525), 36331454fb984af5a6d4f35efe0a2b00 (127.6.204.195:40297)
W20240502 08:02:20.702706 7670 leader_election.cc:341] T ac59612a1fa642f6b7dfd4d899bd6f9d P 16eac9aafbee42bd8a83544ec9b97e74 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d5cb06cedbcb4941afcceab604f34ea4 (127.6.204.193:42525): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:02:20.698081 7827 raft_consensus.cc:348] T cb9b5608afa2415e905426d7400abd6c P d5cb06cedbcb4941afcceab604f34ea4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } }
I20240502 08:02:20.708050 7827 raft_consensus.cc:374] T cb9b5608afa2415e905426d7400abd6c P d5cb06cedbcb4941afcceab604f34ea4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:02:20.708552 7827 raft_consensus.cc:729] T cb9b5608afa2415e905426d7400abd6c P d5cb06cedbcb4941afcceab604f34ea4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5cb06cedbcb4941afcceab604f34ea4, State: Initialized, Role: FOLLOWER
I20240502 08:02:20.709573 7827 consensus_queue.cc:260] T cb9b5608afa2415e905426d7400abd6c P d5cb06cedbcb4941afcceab604f34ea4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } }
I20240502 08:02:20.710502 7781 raft_consensus.cc:2454] T ac59612a1fa642f6b7dfd4d899bd6f9d P 36331454fb984af5a6d4f35efe0a2b00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16eac9aafbee42bd8a83544ec9b97e74 in term 1.
I20240502 08:02:20.707300 7826 tablet_bootstrap.cc:654] T f8a4589c0a9b417188a2d884011b0240 P 36331454fb984af5a6d4f35efe0a2b00: Neither blocks nor log segments found. Creating new log.
I20240502 08:02:20.711647 7857 raft_consensus.cc:482] T cb9b5608afa2415e905426d7400abd6c P d5cb06cedbcb4941afcceab604f34ea4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:02:20.693584 7831 consensus_queue.cc:260] T af8197b1de49493a966b8ba1146ada68 P 16eac9aafbee42bd8a83544ec9b97e74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } }
I20240502 08:02:20.712246 7857 raft_consensus.cc:504] T cb9b5608afa2415e905426d7400abd6c P d5cb06cedbcb4941afcceab604f34ea4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } }
I20240502 08:02:20.713163 7671 leader_election.cc:302] T ac59612a1fa642f6b7dfd4d899bd6f9d P 16eac9aafbee42bd8a83544ec9b97e74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 16eac9aafbee42bd8a83544ec9b97e74, 36331454fb984af5a6d4f35efe0a2b00; no voters: d5cb06cedbcb4941afcceab604f34ea4
I20240502 08:02:20.714241 7834 raft_consensus.cc:2781] T ac59612a1fa642f6b7dfd4d899bd6f9d P 16eac9aafbee42bd8a83544ec9b97e74 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:02:20.714730 7834 raft_consensus.cc:686] T ac59612a1fa642f6b7dfd4d899bd6f9d P 16eac9aafbee42bd8a83544ec9b97e74 [term 1 LEADER]: Becoming Leader. State: Replica: 16eac9aafbee42bd8a83544ec9b97e74, State: Running, Role: LEADER
I20240502 08:02:20.715085 7857 leader_election.cc:288] T cb9b5608afa2415e905426d7400abd6c P d5cb06cedbcb4941afcceab604f34ea4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 16eac9aafbee42bd8a83544ec9b97e74 (127.6.204.194:38399), 36331454fb984af5a6d4f35efe0a2b00 (127.6.204.195:40297)
I20240502 08:02:20.715298 7831 ts_tablet_manager.cc:1398] T af8197b1de49493a966b8ba1146ada68 P 16eac9aafbee42bd8a83544ec9b97e74: Time spent starting tablet: real 0.028s user 0.007s sys 0.000s
I20240502 08:02:20.715581 7834 consensus_queue.cc:237] T ac59612a1fa642f6b7dfd4d899bd6f9d P 16eac9aafbee42bd8a83544ec9b97e74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } }
I20240502 08:02:20.716293 7831 tablet_bootstrap.cc:492] T 8f45c7e5e5cf4b4a84127573dd446bde P 16eac9aafbee42bd8a83544ec9b97e74: Bootstrap starting.
I20240502 08:02:20.716557 7707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb9b5608afa2415e905426d7400abd6c" candidate_uuid: "d5cb06cedbcb4941afcceab604f34ea4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16eac9aafbee42bd8a83544ec9b97e74" is_pre_election: true
I20240502 08:02:20.716908 7827 ts_tablet_manager.cc:1398] T cb9b5608afa2415e905426d7400abd6c P d5cb06cedbcb4941afcceab604f34ea4: Time spent starting tablet: real 0.022s user 0.006s sys 0.002s
I20240502 08:02:20.718784 7827 tablet_bootstrap.cc:492] T f8a4589c0a9b417188a2d884011b0240 P d5cb06cedbcb4941afcceab604f34ea4: Bootstrap starting.
W20240502 08:02:20.724689 7578 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240502 08:02:20.726680 7530 catalog_manager.cc:5484] T cb9b5608afa2415e905426d7400abd6c P 16eac9aafbee42bd8a83544ec9b97e74 reported cstate change: term changed from 0 to 1, leader changed from <none> to 16eac9aafbee42bd8a83544ec9b97e74 (127.6.204.194). New cstate: current_term: 1 leader_uuid: "16eac9aafbee42bd8a83544ec9b97e74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } health_report { overall_health: UNKNOWN } } }
I20240502 08:02:20.728580 7834 raft_consensus.cc:482] T b7961f100fcd49f581e2b1bfa1f1ec05 P 16eac9aafbee42bd8a83544ec9b97e74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:02:20.729270 7827 tablet_bootstrap.cc:654] T f8a4589c0a9b417188a2d884011b0240 P d5cb06cedbcb4941afcceab604f34ea4: Neither blocks nor log segments found. Creating new log.
I20240502 08:02:20.729064 7834 raft_consensus.cc:504] T b7961f100fcd49f581e2b1bfa1f1ec05 P 16eac9aafbee42bd8a83544ec9b97e74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } }
I20240502 08:02:20.733070 7834 leader_election.cc:288] T b7961f100fcd49f581e2b1bfa1f1ec05 P 16eac9aafbee42bd8a83544ec9b97e74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d5cb06cedbcb4941afcceab604f34ea4 (127.6.204.193:42525), 36331454fb984af5a6d4f35efe0a2b00 (127.6.204.195:40297)
I20240502 08:02:20.733858 7632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7961f100fcd49f581e2b1bfa1f1ec05" candidate_uuid: "16eac9aafbee42bd8a83544ec9b97e74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5cb06cedbcb4941afcceab604f34ea4" is_pre_election: true
I20240502 08:02:20.734548 7632 raft_consensus.cc:2454] T b7961f100fcd49f581e2b1bfa1f1ec05 P d5cb06cedbcb4941afcceab604f34ea4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16eac9aafbee42bd8a83544ec9b97e74 in term 0.
I20240502 08:02:20.735803 7780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7961f100fcd49f581e2b1bfa1f1ec05" candidate_uuid: "16eac9aafbee42bd8a83544ec9b97e74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36331454fb984af5a6d4f35efe0a2b00" is_pre_election: true
I20240502 08:02:20.736524 7780 raft_consensus.cc:2454] T b7961f100fcd49f581e2b1bfa1f1ec05 P 36331454fb984af5a6d4f35efe0a2b00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16eac9aafbee42bd8a83544ec9b97e74 in term 0.
I20240502 08:02:20.737669 7781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb9b5608afa2415e905426d7400abd6c" candidate_uuid: "d5cb06cedbcb4941afcceab604f34ea4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36331454fb984af5a6d4f35efe0a2b00" is_pre_election: true
I20240502 08:02:20.739044 7583 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:02:20.740356 7831 tablet_bootstrap.cc:654] T 8f45c7e5e5cf4b4a84127573dd446bde P 16eac9aafbee42bd8a83544ec9b97e74: Neither blocks nor log segments found. Creating new log.
I20240502 08:02:20.740393 7583 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:02:20.738538 7781 raft_consensus.cc:2379] T cb9b5608afa2415e905426d7400abd6c P 36331454fb984af5a6d4f35efe0a2b00 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d5cb06cedbcb4941afcceab604f34ea4 in current term 1: Already voted for candidate 16eac9aafbee42bd8a83544ec9b97e74 in this term.
I20240502 08:02:20.741957 7671 leader_election.cc:302] T b7961f100fcd49f581e2b1bfa1f1ec05 P 16eac9aafbee42bd8a83544ec9b97e74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16eac9aafbee42bd8a83544ec9b97e74, 36331454fb984af5a6d4f35efe0a2b00; no voters:
I20240502 08:02:20.743193 7834 raft_consensus.cc:2781] T b7961f100fcd49f581e2b1bfa1f1ec05 P 16eac9aafbee42bd8a83544ec9b97e74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:02:20.743551 7834 raft_consensus.cc:482] T b7961f100fcd49f581e2b1bfa1f1ec05 P 16eac9aafbee42bd8a83544ec9b97e74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:02:20.743654 7596 leader_election.cc:302] T cb9b5608afa2415e905426d7400abd6c P d5cb06cedbcb4941afcceab604f34ea4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d5cb06cedbcb4941afcceab604f34ea4; no voters: 16eac9aafbee42bd8a83544ec9b97e74, 36331454fb984af5a6d4f35efe0a2b00
I20240502 08:02:20.743942 7834 raft_consensus.cc:3037] T b7961f100fcd49f581e2b1bfa1f1ec05 P 16eac9aafbee42bd8a83544ec9b97e74 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:02:20.744521 7857 raft_consensus.cc:3037] T cb9b5608afa2415e905426d7400abd6c P d5cb06cedbcb4941afcceab604f34ea4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:02:20.746520 7826 tablet_bootstrap.cc:492] T f8a4589c0a9b417188a2d884011b0240 P 36331454fb984af5a6d4f35efe0a2b00: No bootstrap required, opened a new log
I20240502 08:02:20.745739 7528 catalog_manager.cc:5484] T d4042d449ac74cb6bbeb7ba7f90131c4 P d5cb06cedbcb4941afcceab604f34ea4 reported cstate change: term changed from 0 to 1, leader changed from <none> to d5cb06cedbcb4941afcceab604f34ea4 (127.6.204.193). New cstate: current_term: 1 leader_uuid: "d5cb06cedbcb4941afcceab604f34ea4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } health_report { overall_health: UNKNOWN } } }
I20240502 08:02:20.747009 7826 ts_tablet_manager.cc:1367] T f8a4589c0a9b417188a2d884011b0240 P 36331454fb984af5a6d4f35efe0a2b00: Time spent bootstrapping tablet: real 0.071s user 0.003s sys 0.013s
I20240502 08:02:20.748672 7854 raft_consensus.cc:482] T 8e0f7109a1a94eb999a919ed98dea1ce P 16eac9aafbee42bd8a83544ec9b97e74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:02:20.749122 7854 raft_consensus.cc:504] T 8e0f7109a1a94eb999a919ed98dea1ce P 16eac9aafbee42bd8a83544ec9b97e74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } }
I20240502 08:02:20.751665 7854 leader_election.cc:288] T 8e0f7109a1a94eb999a919ed98dea1ce P 16eac9aafbee42bd8a83544ec9b97e74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d5cb06cedbcb4941afcceab604f34ea4 (127.6.204.193:42525), 36331454fb984af5a6d4f35efe0a2b00 (127.6.204.195:40297)
I20240502 08:02:20.752179 7632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e0f7109a1a94eb999a919ed98dea1ce" candidate_uuid: "16eac9aafbee42bd8a83544ec9b97e74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5cb06cedbcb4941afcceab604f34ea4" is_pre_election: true
I20240502 08:02:20.752724 7848 raft_consensus.cc:482] T 8f45c7e5e5cf4b4a84127573dd446bde P d5cb06cedbcb4941afcceab604f34ea4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:02:20.752581 7826 raft_consensus.cc:348] T f8a4589c0a9b417188a2d884011b0240 P 36331454fb984af5a6d4f35efe0a2b00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } }
I20240502 08:02:20.752843 7632 raft_consensus.cc:2454] T 8e0f7109a1a94eb999a919ed98dea1ce P d5cb06cedbcb4941afcceab604f34ea4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16eac9aafbee42bd8a83544ec9b97e74 in term 0.
I20240502 08:02:20.753093 7857 raft_consensus.cc:2726] T cb9b5608afa2415e905426d7400abd6c P d5cb06cedbcb4941afcceab604f34ea4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:02:20.753353 7826 raft_consensus.cc:374] T f8a4589c0a9b417188a2d884011b0240 P 36331454fb984af5a6d4f35efe0a2b00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:02:20.753340 7848 raft_consensus.cc:504] T 8f45c7e5e5cf4b4a84127573dd446bde P d5cb06cedbcb4941afcceab604f34ea4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } }
I20240502 08:02:20.753939 7826 raft_consensus.cc:729] T f8a4589c0a9b417188a2d884011b0240 P 36331454fb984af5a6d4f35efe0a2b00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36331454fb984af5a6d4f35efe0a2b00, State: Initialized, Role: FOLLOWER
I20240502 08:02:20.753837 7834 raft_consensus.cc:504] T b7961f100fcd49f581e2b1bfa1f1ec05 P 16eac9aafbee42bd8a83544ec9b97e74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } }
I20240502 08:02:20.754163 7670 leader_election.cc:302] T 8e0f7109a1a94eb999a919ed98dea1ce P 16eac9aafbee42bd8a83544ec9b97e74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16eac9aafbee42bd8a83544ec9b97e74, d5cb06cedbcb4941afcceab604f34ea4; no voters:
I20240502 08:02:20.755056 7854 raft_consensus.cc:2781] T 8e0f7109a1a94eb999a919ed98dea1ce P 16eac9aafbee42bd8a83544ec9b97e74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:02:20.755398 7854 raft_consensus.cc:482] T 8e0f7109a1a94eb999a919ed98dea1ce P 16eac9aafbee42bd8a83544ec9b97e74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:02:20.755440 7848 leader_election.cc:288] T 8f45c7e5e5cf4b4a84127573dd446bde P d5cb06cedbcb4941afcceab604f34ea4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 16eac9aafbee42bd8a83544ec9b97e74 (127.6.204.194:38399), 36331454fb984af5a6d4f35efe0a2b00 (127.6.204.195:40297)
I20240502 08:02:20.755832 7854 raft_consensus.cc:3037] T 8e0f7109a1a94eb999a919ed98dea1ce P 16eac9aafbee42bd8a83544ec9b97e74 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:02:20.758106 7632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7961f100fcd49f581e2b1bfa1f1ec05" candidate_uuid: "16eac9aafbee42bd8a83544ec9b97e74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5cb06cedbcb4941afcceab604f34ea4"
I20240502 08:02:20.758733 7632 raft_consensus.cc:3037] T b7961f100fcd49f581e2b1bfa1f1ec05 P d5cb06cedbcb4941afcceab604f34ea4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:02:20.759450 7707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f45c7e5e5cf4b4a84127573dd446bde" candidate_uuid: "d5cb06cedbcb4941afcceab604f34ea4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16eac9aafbee42bd8a83544ec9b97e74" is_pre_election: true
I20240502 08:02:20.757162 7826 consensus_queue.cc:260] T f8a4589c0a9b417188a2d884011b0240 P 36331454fb984af5a6d4f35efe0a2b00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } }
I20240502 08:02:20.759799 7781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e0f7109a1a94eb999a919ed98dea1ce" candidate_uuid: "16eac9aafbee42bd8a83544ec9b97e74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36331454fb984af5a6d4f35efe0a2b00" is_pre_election: true
I20240502 08:02:20.758862 7834 leader_election.cc:288] T b7961f100fcd49f581e2b1bfa1f1ec05 P 16eac9aafbee42bd8a83544ec9b97e74 [CANDIDATE]: Term 1 election: Requested vote from peers d5cb06cedbcb4941afcceab604f34ea4 (127.6.204.193:42525), 36331454fb984af5a6d4f35efe0a2b00 (127.6.204.195:40297)
I20240502 08:02:20.763482 7779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f45c7e5e5cf4b4a84127573dd446bde" candidate_uuid: "d5cb06cedbcb4941afcceab604f34ea4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36331454fb984af5a6d4f35efe0a2b00" is_pre_election: true
W20240502 08:02:20.764039 7597 leader_election.cc:341] T 8f45c7e5e5cf4b4a84127573dd446bde P d5cb06cedbcb4941afcceab604f34ea4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 16eac9aafbee42bd8a83544ec9b97e74 (127.6.204.194:38399): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:02:20.765597 7855 raft_consensus.cc:482] T af8197b1de49493a966b8ba1146ada68 P 16eac9aafbee42bd8a83544ec9b97e74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:02:20.766098 7855 raft_consensus.cc:504] T af8197b1de49493a966b8ba1146ada68 P 16eac9aafbee42bd8a83544ec9b97e74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } }
I20240502 08:02:20.767098 7827 tablet_bootstrap.cc:492] T f8a4589c0a9b417188a2d884011b0240 P d5cb06cedbcb4941afcceab604f34ea4: No bootstrap required, opened a new log
I20240502 08:02:20.767443 7826 ts_tablet_manager.cc:1398] T f8a4589c0a9b417188a2d884011b0240 P 36331454fb984af5a6d4f35efe0a2b00: Time spent starting tablet: real 0.020s user 0.007s sys 0.000s
I20240502 08:02:20.767733 7827 ts_tablet_manager.cc:1367] T f8a4589c0a9b417188a2d884011b0240 P d5cb06cedbcb4941afcceab604f34ea4: Time spent bootstrapping tablet: real 0.049s user 0.007s sys 0.007s
I20240502 08:02:20.768225 7781 raft_consensus.cc:2454] T 8e0f7109a1a94eb999a919ed98dea1ce P 36331454fb984af5a6d4f35efe0a2b00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16eac9aafbee42bd8a83544ec9b97e74 in term 0.
I20240502 08:02:20.771596 7827 raft_consensus.cc:348] T f8a4589c0a9b417188a2d884011b0240 P d5cb06cedbcb4941afcceab604f34ea4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } }
I20240502 08:02:20.772635 7631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af8197b1de49493a966b8ba1146ada68" candidate_uuid: "16eac9aafbee42bd8a83544ec9b97e74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5cb06cedbcb4941afcceab604f34ea4" is_pre_election: true
I20240502 08:02:20.772017 7855 leader_election.cc:288] T af8197b1de49493a966b8ba1146ada68 P 16eac9aafbee42bd8a83544ec9b97e74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d5cb06cedbcb4941afcceab604f34ea4 (127.6.204.193:42525), 36331454fb984af5a6d4f35efe0a2b00 (127.6.204.195:40297)
I20240502 08:02:20.777897 7781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af8197b1de49493a966b8ba1146ada68" candidate_uuid: "16eac9aafbee42bd8a83544ec9b97e74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36331454fb984af5a6d4f35efe0a2b00" is_pre_election: true
I20240502 08:02:20.771862 7530 catalog_manager.cc:5484] T ac59612a1fa642f6b7dfd4d899bd6f9d P 16eac9aafbee42bd8a83544ec9b97e74 reported cstate change: term changed from 0 to 1, leader changed from <none> to 16eac9aafbee42bd8a83544ec9b97e74 (127.6.204.194). New cstate: current_term: 1 leader_uuid: "16eac9aafbee42bd8a83544ec9b97e74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } health_report { overall_health: UNKNOWN } } }
I20240502 08:02:20.779105 7781 raft_consensus.cc:2379] T af8197b1de49493a966b8ba1146ada68 P 36331454fb984af5a6d4f35efe0a2b00 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 16eac9aafbee42bd8a83544ec9b97e74 in current term 1: Already voted for candidate d5cb06cedbcb4941afcceab604f34ea4 in this term.
I20240502 08:02:20.780865 7826 tablet_bootstrap.cc:492] T 8f45c7e5e5cf4b4a84127573dd446bde P 36331454fb984af5a6d4f35efe0a2b00: Bootstrap starting.
W20240502 08:02:20.776109 7596 leader_election.cc:341] T 8f45c7e5e5cf4b4a84127573dd446bde P d5cb06cedbcb4941afcceab604f34ea4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 36331454fb984af5a6d4f35efe0a2b00 (127.6.204.195:40297): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:02:20.783164 7596 leader_election.cc:302] T 8f45c7e5e5cf4b4a84127573dd446bde P d5cb06cedbcb4941afcceab604f34ea4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d5cb06cedbcb4941afcceab604f34ea4; no voters: 16eac9aafbee42bd8a83544ec9b97e74, 36331454fb984af5a6d4f35efe0a2b00
I20240502 08:02:20.783910 7848 raft_consensus.cc:2726] T 8f45c7e5e5cf4b4a84127573dd446bde P d5cb06cedbcb4941afcceab604f34ea4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:02:20.777094 7780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7961f100fcd49f581e2b1bfa1f1ec05" candidate_uuid: "16eac9aafbee42bd8a83544ec9b97e74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36331454fb984af5a6d4f35efe0a2b00"
I20240502 08:02:20.784745 7780 raft_consensus.cc:3037] T b7961f100fcd49f581e2b1bfa1f1ec05 P 36331454fb984af5a6d4f35efe0a2b00 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:02:20.773229 7827 raft_consensus.cc:374] T f8a4589c0a9b417188a2d884011b0240 P d5cb06cedbcb4941afcceab604f34ea4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:02:20.789642 7827 raft_consensus.cc:729] T f8a4589c0a9b417188a2d884011b0240 P d5cb06cedbcb4941afcceab604f34ea4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5cb06cedbcb4941afcceab604f34ea4, State: Initialized, Role: FOLLOWER
I20240502 08:02:20.791524 7670 leader_election.cc:302] T af8197b1de49493a966b8ba1146ada68 P 16eac9aafbee42bd8a83544ec9b97e74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 16eac9aafbee42bd8a83544ec9b97e74; no voters: 36331454fb984af5a6d4f35efe0a2b00, d5cb06cedbcb4941afcceab604f34ea4
I20240502 08:02:20.793037 7855 raft_consensus.cc:3037] T af8197b1de49493a966b8ba1146ada68 P 16eac9aafbee42bd8a83544ec9b97e74 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:02:20.796095 7827 consensus_queue.cc:260] T f8a4589c0a9b417188a2d884011b0240 P d5cb06cedbcb4941afcceab604f34ea4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } }
I20240502 08:02:20.798117 7854 raft_consensus.cc:504] T 8e0f7109a1a94eb999a919ed98dea1ce P 16eac9aafbee42bd8a83544ec9b97e74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } }
I20240502 08:02:20.798856 7632 raft_consensus.cc:2454] T b7961f100fcd49f581e2b1bfa1f1ec05 P d5cb06cedbcb4941afcceab604f34ea4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16eac9aafbee42bd8a83544ec9b97e74 in term 1.
I20240502 08:02:20.800009 7670 leader_election.cc:302] T b7961f100fcd49f581e2b1bfa1f1ec05 P 16eac9aafbee42bd8a83544ec9b97e74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16eac9aafbee42bd8a83544ec9b97e74, d5cb06cedbcb4941afcceab604f34ea4; no voters:
I20240502 08:02:20.800415 7854 leader_election.cc:288] T 8e0f7109a1a94eb999a919ed98dea1ce P 16eac9aafbee42bd8a83544ec9b97e74 [CANDIDATE]: Term 1 election: Requested vote from peers d5cb06cedbcb4941afcceab604f34ea4 (127.6.204.193:42525), 36331454fb984af5a6d4f35efe0a2b00 (127.6.204.195:40297)
I20240502 08:02:20.800920 7854 raft_consensus.cc:2781] T b7961f100fcd49f581e2b1bfa1f1ec05 P 16eac9aafbee42bd8a83544ec9b97e74 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:02:20.801373 7854 raft_consensus.cc:686] T b7961f100fcd49f581e2b1bfa1f1ec05 P 16eac9aafbee42bd8a83544ec9b97e74 [term 1 LEADER]: Becoming Leader. State: Replica: 16eac9aafbee42bd8a83544ec9b97e74, State: Running, Role: LEADER
I20240502 08:02:20.801707 7631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e0f7109a1a94eb999a919ed98dea1ce" candidate_uuid: "16eac9aafbee42bd8a83544ec9b97e74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5cb06cedbcb4941afcceab604f34ea4"
I20240502 08:02:20.802384 7631 raft_consensus.cc:3037] T 8e0f7109a1a94eb999a919ed98dea1ce P d5cb06cedbcb4941afcceab604f34ea4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:02:20.802034 7781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e0f7109a1a94eb999a919ed98dea1ce" candidate_uuid: "16eac9aafbee42bd8a83544ec9b97e74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36331454fb984af5a6d4f35efe0a2b00"
I20240502 08:02:20.804728 7781 raft_consensus.cc:3037] T 8e0f7109a1a94eb999a919ed98dea1ce P 36331454fb984af5a6d4f35efe0a2b00 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:02:20.803205 7854 consensus_queue.cc:237] T b7961f100fcd49f581e2b1bfa1f1ec05 P 16eac9aafbee42bd8a83544ec9b97e74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } }
I20240502 08:02:20.812462 7855 raft_consensus.cc:2726] T af8197b1de49493a966b8ba1146ada68 P 16eac9aafbee42bd8a83544ec9b97e74 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:02:20.815917 7833 raft_consensus.cc:482] T f8a4589c0a9b417188a2d884011b0240 P 36331454fb984af5a6d4f35efe0a2b00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:02:20.817056 7831 tablet_bootstrap.cc:492] T 8f45c7e5e5cf4b4a84127573dd446bde P 16eac9aafbee42bd8a83544ec9b97e74: No bootstrap required, opened a new log
I20240502 08:02:20.816397 7833 raft_consensus.cc:504] T f8a4589c0a9b417188a2d884011b0240 P 36331454fb984af5a6d4f35efe0a2b00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } }
I20240502 08:02:20.809836 7780 raft_consensus.cc:2454] T b7961f100fcd49f581e2b1bfa1f1ec05 P 36331454fb984af5a6d4f35efe0a2b00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16eac9aafbee42bd8a83544ec9b97e74 in term 1.
I20240502 08:02:20.817548 7831 ts_tablet_manager.cc:1367] T 8f45c7e5e5cf4b4a84127573dd446bde P 16eac9aafbee42bd8a83544ec9b97e74: Time spent bootstrapping tablet: real 0.101s user 0.015s sys 0.015s
I20240502 08:02:20.820385 7631 raft_consensus.cc:2454] T 8e0f7109a1a94eb999a919ed98dea1ce P d5cb06cedbcb4941afcceab604f34ea4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16eac9aafbee42bd8a83544ec9b97e74 in term 1.
I20240502 08:02:20.827035 7670 leader_election.cc:302] T 8e0f7109a1a94eb999a919ed98dea1ce P 16eac9aafbee42bd8a83544ec9b97e74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16eac9aafbee42bd8a83544ec9b97e74, d5cb06cedbcb4941afcceab604f34ea4; no voters:
I20240502 08:02:20.830551 7833 leader_election.cc:288] T f8a4589c0a9b417188a2d884011b0240 P 36331454fb984af5a6d4f35efe0a2b00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d5cb06cedbcb4941afcceab604f34ea4 (127.6.204.193:42525), 16eac9aafbee42bd8a83544ec9b97e74 (127.6.204.194:38399)
I20240502 08:02:20.839053 7831 raft_consensus.cc:348] T 8f45c7e5e5cf4b4a84127573dd446bde P 16eac9aafbee42bd8a83544ec9b97e74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } }
I20240502 08:02:20.840093 7831 raft_consensus.cc:374] T 8f45c7e5e5cf4b4a84127573dd446bde P 16eac9aafbee42bd8a83544ec9b97e74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:02:20.840440 7831 raft_consensus.cc:729] T 8f45c7e5e5cf4b4a84127573dd446bde P 16eac9aafbee42bd8a83544ec9b97e74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16eac9aafbee42bd8a83544ec9b97e74, State: Initialized, Role: FOLLOWER
I20240502 08:02:20.846030 7855 raft_consensus.cc:2781] T 8e0f7109a1a94eb999a919ed98dea1ce P 16eac9aafbee42bd8a83544ec9b97e74 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:02:20.847326 7855 raft_consensus.cc:686] T 8e0f7109a1a94eb999a919ed98dea1ce P 16eac9aafbee42bd8a83544ec9b97e74 [term 1 LEADER]: Becoming Leader. State: Replica: 16eac9aafbee42bd8a83544ec9b97e74, State: Running, Role: LEADER
I20240502 08:02:20.841300 7831 consensus_queue.cc:260] T 8f45c7e5e5cf4b4a84127573dd446bde P 16eac9aafbee42bd8a83544ec9b97e74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } }
I20240502 08:02:20.850381 7831 ts_tablet_manager.cc:1398] T 8f45c7e5e5cf4b4a84127573dd446bde P 16eac9aafbee42bd8a83544ec9b97e74: Time spent starting tablet: real 0.030s user 0.007s sys 0.001s
I20240502 08:02:20.849220 7529 catalog_manager.cc:5484] T b7961f100fcd49f581e2b1bfa1f1ec05 P 16eac9aafbee42bd8a83544ec9b97e74 reported cstate change: term changed from 0 to 1, leader changed from <none> to 16eac9aafbee42bd8a83544ec9b97e74 (127.6.204.194). New cstate: current_term: 1 leader_uuid: "16eac9aafbee42bd8a83544ec9b97e74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } health_report { overall_health: UNKNOWN } } }
I20240502 08:02:20.848395 7855 consensus_queue.cc:237] T 8e0f7109a1a94eb999a919ed98dea1ce P 16eac9aafbee42bd8a83544ec9b97e74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } }
I20240502 08:02:20.859114 7826 tablet_bootstrap.cc:654] T 8f45c7e5e5cf4b4a84127573dd446bde P 36331454fb984af5a6d4f35efe0a2b00: Neither blocks nor log segments found. Creating new log.
I20240502 08:02:20.860982 7781 raft_consensus.cc:2454] T 8e0f7109a1a94eb999a919ed98dea1ce P 36331454fb984af5a6d4f35efe0a2b00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16eac9aafbee42bd8a83544ec9b97e74 in term 1.
I20240502 08:02:20.865631 7827 ts_tablet_manager.cc:1398] T f8a4589c0a9b417188a2d884011b0240 P d5cb06cedbcb4941afcceab604f34ea4: Time spent starting tablet: real 0.097s user 0.006s sys 0.004s
I20240502 08:02:20.869421 7827 tablet_bootstrap.cc:492] T ac59612a1fa642f6b7dfd4d899bd6f9d P d5cb06cedbcb4941afcceab604f34ea4: Bootstrap starting.
I20240502 08:02:20.871311 7528 catalog_manager.cc:5484] T 8e0f7109a1a94eb999a919ed98dea1ce P 16eac9aafbee42bd8a83544ec9b97e74 reported cstate change: term changed from 0 to 1, leader changed from <none> to 16eac9aafbee42bd8a83544ec9b97e74 (127.6.204.194). New cstate: current_term: 1 leader_uuid: "16eac9aafbee42bd8a83544ec9b97e74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } health_report { overall_health: UNKNOWN } } }
I20240502 08:02:20.875079 7827 tablet_bootstrap.cc:654] T ac59612a1fa642f6b7dfd4d899bd6f9d P d5cb06cedbcb4941afcceab604f34ea4: Neither blocks nor log segments found. Creating new log.
I20240502 08:02:20.875895 7631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8a4589c0a9b417188a2d884011b0240" candidate_uuid: "36331454fb984af5a6d4f35efe0a2b00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5cb06cedbcb4941afcceab604f34ea4" is_pre_election: true
I20240502 08:02:20.876631 7631 raft_consensus.cc:2454] T f8a4589c0a9b417188a2d884011b0240 P d5cb06cedbcb4941afcceab604f34ea4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36331454fb984af5a6d4f35efe0a2b00 in term 0.
I20240502 08:02:20.878226 7744 leader_election.cc:302] T f8a4589c0a9b417188a2d884011b0240 P 36331454fb984af5a6d4f35efe0a2b00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36331454fb984af5a6d4f35efe0a2b00, d5cb06cedbcb4941afcceab604f34ea4; no voters:
I20240502 08:02:20.879392 7833 raft_consensus.cc:2781] T f8a4589c0a9b417188a2d884011b0240 P 36331454fb984af5a6d4f35efe0a2b00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:02:20.879771 7833 raft_consensus.cc:482] T f8a4589c0a9b417188a2d884011b0240 P 36331454fb984af5a6d4f35efe0a2b00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:02:20.880100 7833 raft_consensus.cc:3037] T f8a4589c0a9b417188a2d884011b0240 P 36331454fb984af5a6d4f35efe0a2b00 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:02:20.884702 7826 tablet_bootstrap.cc:492] T 8f45c7e5e5cf4b4a84127573dd446bde P 36331454fb984af5a6d4f35efe0a2b00: No bootstrap required, opened a new log
I20240502 08:02:20.885322 7826 ts_tablet_manager.cc:1367] T 8f45c7e5e5cf4b4a84127573dd446bde P 36331454fb984af5a6d4f35efe0a2b00: Time spent bootstrapping tablet: real 0.105s user 0.012s sys 0.002s
I20240502 08:02:20.887156 7833 raft_consensus.cc:504] T f8a4589c0a9b417188a2d884011b0240 P 36331454fb984af5a6d4f35efe0a2b00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } }
I20240502 08:02:20.888671 7826 raft_consensus.cc:348] T 8f45c7e5e5cf4b4a84127573dd446bde P 36331454fb984af5a6d4f35efe0a2b00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } }
I20240502 08:02:20.889330 7827 tablet_bootstrap.cc:492] T ac59612a1fa642f6b7dfd4d899bd6f9d P d5cb06cedbcb4941afcceab604f34ea4: No bootstrap required, opened a new log
I20240502 08:02:20.889446 7826 raft_consensus.cc:374] T 8f45c7e5e5cf4b4a84127573dd446bde P 36331454fb984af5a6d4f35efe0a2b00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:02:20.889806 7826 raft_consensus.cc:729] T 8f45c7e5e5cf4b4a84127573dd446bde P 36331454fb984af5a6d4f35efe0a2b00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36331454fb984af5a6d4f35efe0a2b00, State: Initialized, Role: FOLLOWER
I20240502 08:02:20.889912 7827 ts_tablet_manager.cc:1367] T ac59612a1fa642f6b7dfd4d899bd6f9d P d5cb06cedbcb4941afcceab604f34ea4: Time spent bootstrapping tablet: real 0.021s user 0.010s sys 0.007s
I20240502 08:02:20.890537 7631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8a4589c0a9b417188a2d884011b0240" candidate_uuid: "36331454fb984af5a6d4f35efe0a2b00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5cb06cedbcb4941afcceab604f34ea4"
I20240502 08:02:20.891165 7631 raft_consensus.cc:3037] T f8a4589c0a9b417188a2d884011b0240 P d5cb06cedbcb4941afcceab604f34ea4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:02:20.890751 7826 consensus_queue.cc:260] T 8f45c7e5e5cf4b4a84127573dd446bde P 36331454fb984af5a6d4f35efe0a2b00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } }
I20240502 08:02:20.894243 7707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8a4589c0a9b417188a2d884011b0240" candidate_uuid: "36331454fb984af5a6d4f35efe0a2b00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16eac9aafbee42bd8a83544ec9b97e74" is_pre_election: true
I20240502 08:02:20.894704 7706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8a4589c0a9b417188a2d884011b0240" candidate_uuid: "36331454fb984af5a6d4f35efe0a2b00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16eac9aafbee42bd8a83544ec9b97e74"
I20240502 08:02:20.895134 7707 raft_consensus.cc:2454] T f8a4589c0a9b417188a2d884011b0240 P 16eac9aafbee42bd8a83544ec9b97e74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36331454fb984af5a6d4f35efe0a2b00 in term 0.
I20240502 08:02:20.893208 7827 raft_consensus.cc:348] T ac59612a1fa642f6b7dfd4d899bd6f9d P d5cb06cedbcb4941afcceab604f34ea4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } }
I20240502 08:02:20.896610 7827 raft_consensus.cc:374] T ac59612a1fa642f6b7dfd4d899bd6f9d P d5cb06cedbcb4941afcceab604f34ea4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:02:20.897015 7827 raft_consensus.cc:729] T ac59612a1fa642f6b7dfd4d899bd6f9d P d5cb06cedbcb4941afcceab604f34ea4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5cb06cedbcb4941afcceab604f34ea4, State: Initialized, Role: FOLLOWER
I20240502 08:02:20.897948 7631 raft_consensus.cc:2454] T f8a4589c0a9b417188a2d884011b0240 P d5cb06cedbcb4941afcceab604f34ea4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36331454fb984af5a6d4f35efe0a2b00 in term 1.
I20240502 08:02:20.897891 7827 consensus_queue.cc:260] T ac59612a1fa642f6b7dfd4d899bd6f9d P d5cb06cedbcb4941afcceab604f34ea4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } }
I20240502 08:02:20.899245 7744 leader_election.cc:302] T f8a4589c0a9b417188a2d884011b0240 P 36331454fb984af5a6d4f35efe0a2b00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 36331454fb984af5a6d4f35efe0a2b00, d5cb06cedbcb4941afcceab604f34ea4; no voters: 16eac9aafbee42bd8a83544ec9b97e74
I20240502 08:02:20.900180 7833 leader_election.cc:288] T f8a4589c0a9b417188a2d884011b0240 P 36331454fb984af5a6d4f35efe0a2b00 [CANDIDATE]: Term 1 election: Requested vote from peers d5cb06cedbcb4941afcceab604f34ea4 (127.6.204.193:42525), 16eac9aafbee42bd8a83544ec9b97e74 (127.6.204.194:38399)
I20240502 08:02:20.901476 7877 raft_consensus.cc:2781] T f8a4589c0a9b417188a2d884011b0240 P 36331454fb984af5a6d4f35efe0a2b00 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:02:20.902029 7877 raft_consensus.cc:686] T f8a4589c0a9b417188a2d884011b0240 P 36331454fb984af5a6d4f35efe0a2b00 [term 1 LEADER]: Becoming Leader. State: Replica: 36331454fb984af5a6d4f35efe0a2b00, State: Running, Role: LEADER
I20240502 08:02:20.903584 7877 consensus_queue.cc:237] T f8a4589c0a9b417188a2d884011b0240 P 36331454fb984af5a6d4f35efe0a2b00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } }
I20240502 08:02:20.905993 7833 raft_consensus.cc:482] T 8f45c7e5e5cf4b4a84127573dd446bde P 36331454fb984af5a6d4f35efe0a2b00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:02:20.906548 7833 raft_consensus.cc:504] T 8f45c7e5e5cf4b4a84127573dd446bde P 36331454fb984af5a6d4f35efe0a2b00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } }
I20240502 08:02:20.909818 7631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f45c7e5e5cf4b4a84127573dd446bde" candidate_uuid: "36331454fb984af5a6d4f35efe0a2b00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5cb06cedbcb4941afcceab604f34ea4" is_pre_election: true
I20240502 08:02:20.910650 7833 leader_election.cc:288] T 8f45c7e5e5cf4b4a84127573dd446bde P 36331454fb984af5a6d4f35efe0a2b00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d5cb06cedbcb4941afcceab604f34ea4 (127.6.204.193:42525), 16eac9aafbee42bd8a83544ec9b97e74 (127.6.204.194:38399)
I20240502 08:02:20.910501 7631 raft_consensus.cc:2454] T 8f45c7e5e5cf4b4a84127573dd446bde P d5cb06cedbcb4941afcceab604f34ea4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36331454fb984af5a6d4f35efe0a2b00 in term 0.
I20240502 08:02:20.912936 7744 leader_election.cc:302] T 8f45c7e5e5cf4b4a84127573dd446bde P 36331454fb984af5a6d4f35efe0a2b00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36331454fb984af5a6d4f35efe0a2b00, d5cb06cedbcb4941afcceab604f34ea4; no voters:
I20240502 08:02:20.913758 7833 raft_consensus.cc:2781] T 8f45c7e5e5cf4b4a84127573dd446bde P 36331454fb984af5a6d4f35efe0a2b00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:02:20.910532 7707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f45c7e5e5cf4b4a84127573dd446bde" candidate_uuid: "36331454fb984af5a6d4f35efe0a2b00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16eac9aafbee42bd8a83544ec9b97e74" is_pre_election: true
I20240502 08:02:20.914042 7833 raft_consensus.cc:482] T 8f45c7e5e5cf4b4a84127573dd446bde P 36331454fb984af5a6d4f35efe0a2b00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:02:20.914170 7827 ts_tablet_manager.cc:1398] T ac59612a1fa642f6b7dfd4d899bd6f9d P d5cb06cedbcb4941afcceab604f34ea4: Time spent starting tablet: real 0.024s user 0.006s sys 0.000s
I20240502 08:02:20.914412 7833 raft_consensus.cc:3037] T 8f45c7e5e5cf4b4a84127573dd446bde P 36331454fb984af5a6d4f35efe0a2b00 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:02:20.914474 7707 raft_consensus.cc:2454] T 8f45c7e5e5cf4b4a84127573dd446bde P 16eac9aafbee42bd8a83544ec9b97e74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36331454fb984af5a6d4f35efe0a2b00 in term 0.
I20240502 08:02:20.916198 7826 ts_tablet_manager.cc:1398] T 8f45c7e5e5cf4b4a84127573dd446bde P 36331454fb984af5a6d4f35efe0a2b00: Time spent starting tablet: real 0.030s user 0.022s sys 0.006s
I20240502 08:02:20.919488 7833 raft_consensus.cc:504] T 8f45c7e5e5cf4b4a84127573dd446bde P 36331454fb984af5a6d4f35efe0a2b00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } }
I20240502 08:02:20.921675 7833 leader_election.cc:288] T 8f45c7e5e5cf4b4a84127573dd446bde P 36331454fb984af5a6d4f35efe0a2b00 [CANDIDATE]: Term 1 election: Requested vote from peers d5cb06cedbcb4941afcceab604f34ea4 (127.6.204.193:42525), 16eac9aafbee42bd8a83544ec9b97e74 (127.6.204.194:38399)
I20240502 08:02:20.921940 7631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f45c7e5e5cf4b4a84127573dd446bde" candidate_uuid: "36331454fb984af5a6d4f35efe0a2b00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5cb06cedbcb4941afcceab604f34ea4"
I20240502 08:02:20.922472 7707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f45c7e5e5cf4b4a84127573dd446bde" candidate_uuid: "36331454fb984af5a6d4f35efe0a2b00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16eac9aafbee42bd8a83544ec9b97e74"
I20240502 08:02:20.922933 7707 raft_consensus.cc:3037] T 8f45c7e5e5cf4b4a84127573dd446bde P 16eac9aafbee42bd8a83544ec9b97e74 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:02:20.922919 7631 raft_consensus.cc:3037] T 8f45c7e5e5cf4b4a84127573dd446bde P d5cb06cedbcb4941afcceab604f34ea4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:02:20.924870 7528 catalog_manager.cc:5484] T f8a4589c0a9b417188a2d884011b0240 P 36331454fb984af5a6d4f35efe0a2b00 reported cstate change: term changed from 0 to 1, leader changed from <none> to 36331454fb984af5a6d4f35efe0a2b00 (127.6.204.195). New cstate: current_term: 1 leader_uuid: "36331454fb984af5a6d4f35efe0a2b00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } health_report { overall_health: HEALTHY } } }
I20240502 08:02:20.927778 7707 raft_consensus.cc:2454] T 8f45c7e5e5cf4b4a84127573dd446bde P 16eac9aafbee42bd8a83544ec9b97e74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36331454fb984af5a6d4f35efe0a2b00 in term 1.
I20240502 08:02:20.929023 7746 leader_election.cc:302] T 8f45c7e5e5cf4b4a84127573dd446bde P 36331454fb984af5a6d4f35efe0a2b00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16eac9aafbee42bd8a83544ec9b97e74, 36331454fb984af5a6d4f35efe0a2b00; no voters:
I20240502 08:02:20.929353 7631 raft_consensus.cc:2454] T 8f45c7e5e5cf4b4a84127573dd446bde P d5cb06cedbcb4941afcceab604f34ea4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36331454fb984af5a6d4f35efe0a2b00 in term 1.
I20240502 08:02:20.929939 7833 raft_consensus.cc:2781] T 8f45c7e5e5cf4b4a84127573dd446bde P 36331454fb984af5a6d4f35efe0a2b00 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:02:20.930495 7833 raft_consensus.cc:686] T 8f45c7e5e5cf4b4a84127573dd446bde P 36331454fb984af5a6d4f35efe0a2b00 [term 1 LEADER]: Becoming Leader. State: Replica: 36331454fb984af5a6d4f35efe0a2b00, State: Running, Role: LEADER
I20240502 08:02:20.931283 7833 consensus_queue.cc:237] T 8f45c7e5e5cf4b4a84127573dd446bde P 36331454fb984af5a6d4f35efe0a2b00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } }
I20240502 08:02:20.938443 7528 catalog_manager.cc:5484] T 8f45c7e5e5cf4b4a84127573dd446bde P 36331454fb984af5a6d4f35efe0a2b00 reported cstate change: term changed from 0 to 1, leader changed from <none> to 36331454fb984af5a6d4f35efe0a2b00 (127.6.204.195). New cstate: current_term: 1 leader_uuid: "36331454fb984af5a6d4f35efe0a2b00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } health_report { overall_health: HEALTHY } } }
I20240502 08:02:21.007918 6963 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:02:21.013603 7880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:02:21.014709 7881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:02:21.015633 7883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:02:21.016158 6963 server_base.cc:1034] running on GCE node
I20240502 08:02:21.017019 6963 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:02:21.017218 6963 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:02:21.017400 6963 hybrid_clock.cc:648] HybridClock initialized: now 1714636941017384 us; error 0 us; skew 500 ppm
I20240502 08:02:21.017925 6963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:02:21.020578 6963 webserver.cc:460] Webserver started at http://127.6.204.196:42681/ using document root <none> and password file <none>
I20240502 08:02:21.021121 6963 fs_manager.cc:359] Metadata directory not provided
I20240502 08:02:21.021342 6963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:02:21.021608 6963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:02:21.022814 6963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714636933625983-6963-0/minicluster-data/ts-3-root/instance:
uuid: "89fb59c0fd364e37b3486a7b04ed9f4c"
format_stamp: "Formatted at 2024-05-02 08:02:21 on dist-test-slave-32xs"
I20240502 08:02:21.027634 6963 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 08:02:21.028234 7848 consensus_queue.cc:1035] T af8197b1de49493a966b8ba1146ada68 P d5cb06cedbcb4941afcceab604f34ea4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:02:21.031390 7888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:02:21.032564 6963 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240502 08:02:21.033056 6963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714636933625983-6963-0/minicluster-data/ts-3-root
uuid: "89fb59c0fd364e37b3486a7b04ed9f4c"
format_stamp: "Formatted at 2024-05-02 08:02:21 on dist-test-slave-32xs"
I20240502 08:02:21.033928 6963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714636933625983-6963-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714636933625983-6963-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714636933625983-6963-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:02:21.042526 7857 consensus_queue.cc:1035] T af8197b1de49493a966b8ba1146ada68 P d5cb06cedbcb4941afcceab604f34ea4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:02:21.051357 7855 consensus_queue.cc:1035] T cb9b5608afa2415e905426d7400abd6c P 16eac9aafbee42bd8a83544ec9b97e74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:02:21.062281 7854 consensus_queue.cc:1035] T cb9b5608afa2415e905426d7400abd6c P 16eac9aafbee42bd8a83544ec9b97e74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:02:21.088306 6963 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:02:21.089372 6963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:02:21.090907 6963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:02:21.093204 6963 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:02:21.093402 6963 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:02:21.093636 6963 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:02:21.093858 6963 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:02:21.105243 7857 raft_consensus.cc:482] T ac59612a1fa642f6b7dfd4d899bd6f9d P d5cb06cedbcb4941afcceab604f34ea4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:02:21.105733 7857 raft_consensus.cc:504] T ac59612a1fa642f6b7dfd4d899bd6f9d P d5cb06cedbcb4941afcceab604f34ea4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } }
I20240502 08:02:21.107733 7857 leader_election.cc:288] T ac59612a1fa642f6b7dfd4d899bd6f9d P d5cb06cedbcb4941afcceab604f34ea4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 16eac9aafbee42bd8a83544ec9b97e74 (127.6.204.194:38399), 36331454fb984af5a6d4f35efe0a2b00 (127.6.204.195:40297)
I20240502 08:02:21.109397 7781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac59612a1fa642f6b7dfd4d899bd6f9d" candidate_uuid: "d5cb06cedbcb4941afcceab604f34ea4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36331454fb984af5a6d4f35efe0a2b00" is_pre_election: true
I20240502 08:02:21.110291 7781 raft_consensus.cc:2379] T ac59612a1fa642f6b7dfd4d899bd6f9d P 36331454fb984af5a6d4f35efe0a2b00 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d5cb06cedbcb4941afcceab604f34ea4 in current term 1: Already voted for candidate 16eac9aafbee42bd8a83544ec9b97e74 in this term.
I20240502 08:02:21.110436 7707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac59612a1fa642f6b7dfd4d899bd6f9d" candidate_uuid: "d5cb06cedbcb4941afcceab604f34ea4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16eac9aafbee42bd8a83544ec9b97e74" is_pre_election: true
I20240502 08:02:21.112129 7597 leader_election.cc:302] T ac59612a1fa642f6b7dfd4d899bd6f9d P d5cb06cedbcb4941afcceab604f34ea4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d5cb06cedbcb4941afcceab604f34ea4; no voters: 16eac9aafbee42bd8a83544ec9b97e74, 36331454fb984af5a6d4f35efe0a2b00
I20240502 08:02:21.113103 7857 raft_consensus.cc:3037] T ac59612a1fa642f6b7dfd4d899bd6f9d P d5cb06cedbcb4941afcceab604f34ea4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:02:21.117621 7857 raft_consensus.cc:2726] T ac59612a1fa642f6b7dfd4d899bd6f9d P d5cb06cedbcb4941afcceab604f34ea4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:02:21.139214 6963 rpc_server.cc:307] RPC server started. Bound to: 127.6.204.196:36987
I20240502 08:02:21.139317 7958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.204.196:36987 every 8 connection(s)
I20240502 08:02:21.152606 7959 heartbeater.cc:347] Connected to a master server at 127.6.204.254:40853
I20240502 08:02:21.152967 7959 heartbeater.cc:464] Registering TS with master...
I20240502 08:02:21.153637 7959 heartbeater.cc:511] Master 127.6.204.254:40853 requested a full tablet report, sending...
I20240502 08:02:21.155380 7529 ts_manager.cc:194] Registered new tserver with Master: 89fb59c0fd364e37b3486a7b04ed9f4c (127.6.204.196:36987)
I20240502 08:02:21.156610 7529 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48276
I20240502 08:02:21.201059 7854 consensus_queue.cc:1035] T ac59612a1fa642f6b7dfd4d899bd6f9d P 16eac9aafbee42bd8a83544ec9b97e74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:02:21.207341 7854 consensus_queue.cc:1035] T b7961f100fcd49f581e2b1bfa1f1ec05 P 16eac9aafbee42bd8a83544ec9b97e74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:02:21.211249 7855 consensus_queue.cc:1035] T ac59612a1fa642f6b7dfd4d899bd6f9d P 16eac9aafbee42bd8a83544ec9b97e74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:02:21.216460 7855 consensus_queue.cc:1035] T b7961f100fcd49f581e2b1bfa1f1ec05 P 16eac9aafbee42bd8a83544ec9b97e74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:02:21.226059 7830 consensus_queue.cc:1035] T d4042d449ac74cb6bbeb7ba7f90131c4 P d5cb06cedbcb4941afcceab604f34ea4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:02:21.244482 7830 consensus_queue.cc:1035] T d4042d449ac74cb6bbeb7ba7f90131c4 P d5cb06cedbcb4941afcceab604f34ea4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:02:21.328583 7833 consensus_queue.cc:1035] T 8f45c7e5e5cf4b4a84127573dd446bde P 36331454fb984af5a6d4f35efe0a2b00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:02:21.339619 7854 consensus_queue.cc:1035] T 8e0f7109a1a94eb999a919ed98dea1ce P 16eac9aafbee42bd8a83544ec9b97e74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:02:21.340145 7876 consensus_queue.cc:1035] T 8f45c7e5e5cf4b4a84127573dd446bde P 36331454fb984af5a6d4f35efe0a2b00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 08:02:21.345248 7854 consensus_queue.cc:1035] T 8e0f7109a1a94eb999a919ed98dea1ce P 16eac9aafbee42bd8a83544ec9b97e74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 08:02:21.493897 7707 raft_consensus.cc:3037] T f8a4589c0a9b417188a2d884011b0240 P 16eac9aafbee42bd8a83544ec9b97e74 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:02:21.497748 7876 consensus_queue.cc:1035] T f8a4589c0a9b417188a2d884011b0240 P 36331454fb984af5a6d4f35efe0a2b00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:02:21.500335 7876 consensus_queue.cc:1035] T f8a4589c0a9b417188a2d884011b0240 P 36331454fb984af5a6d4f35efe0a2b00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:02:22.158684 7959 heartbeater.cc:503] Master 127.6.204.254:40853 was elected leader, sending a full tablet report...
I20240502 08:02:22.743398 7583 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:02:22.759125 7707 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cb9b5608afa2415e905426d7400abd6c"
dest_uuid: "16eac9aafbee42bd8a83544ec9b97e74"
mode: GRACEFUL
new_leader_uuid: "d5cb06cedbcb4941afcceab604f34ea4"
from {username='slave'} at 127.0.0.1:43416
I20240502 08:02:22.759755 7707 raft_consensus.cc:595] T cb9b5608afa2415e905426d7400abd6c P 16eac9aafbee42bd8a83544ec9b97e74 [term 1 LEADER]: Received request to transfer leadership to TS d5cb06cedbcb4941afcceab604f34ea4
I20240502 08:02:22.760872 7583 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240502 08:02:22.761373 7583 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:02:23.212649 7834 raft_consensus.cc:979] T cb9b5608afa2415e905426d7400abd6c P 16eac9aafbee42bd8a83544ec9b97e74: : Instructing follower d5cb06cedbcb4941afcceab604f34ea4 to start an election
I20240502 08:02:23.213104 7979 raft_consensus.cc:1067] T cb9b5608afa2415e905426d7400abd6c P 16eac9aafbee42bd8a83544ec9b97e74 [term 1 LEADER]: Signalling peer d5cb06cedbcb4941afcceab604f34ea4 to start an election
I20240502 08:02:23.214809 7632 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cb9b5608afa2415e905426d7400abd6c"
dest_uuid: "d5cb06cedbcb4941afcceab604f34ea4"
from {username='slave'} at 127.0.0.1:55526
I20240502 08:02:23.215400 7632 raft_consensus.cc:482] T cb9b5608afa2415e905426d7400abd6c P d5cb06cedbcb4941afcceab604f34ea4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:02:23.215719 7632 raft_consensus.cc:3037] T cb9b5608afa2415e905426d7400abd6c P d5cb06cedbcb4941afcceab604f34ea4 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:02:23.220453 7632 raft_consensus.cc:504] T cb9b5608afa2415e905426d7400abd6c P d5cb06cedbcb4941afcceab604f34ea4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } }
I20240502 08:02:23.222084 7632 leader_election.cc:288] T cb9b5608afa2415e905426d7400abd6c P d5cb06cedbcb4941afcceab604f34ea4 [CANDIDATE]: Term 2 election: Requested vote from peers 16eac9aafbee42bd8a83544ec9b97e74 (127.6.204.194:38399), 36331454fb984af5a6d4f35efe0a2b00 (127.6.204.195:40297)
I20240502 08:02:23.222931 7707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb9b5608afa2415e905426d7400abd6c" candidate_uuid: "d5cb06cedbcb4941afcceab604f34ea4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "16eac9aafbee42bd8a83544ec9b97e74"
I20240502 08:02:23.223186 7780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb9b5608afa2415e905426d7400abd6c" candidate_uuid: "d5cb06cedbcb4941afcceab604f34ea4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "36331454fb984af5a6d4f35efe0a2b00"
I20240502 08:02:23.223464 7707 raft_consensus.cc:3032] T cb9b5608afa2415e905426d7400abd6c P 16eac9aafbee42bd8a83544ec9b97e74 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:02:23.223757 7780 raft_consensus.cc:3037] T cb9b5608afa2415e905426d7400abd6c P 36331454fb984af5a6d4f35efe0a2b00 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:02:23.223783 7707 raft_consensus.cc:729] T cb9b5608afa2415e905426d7400abd6c P 16eac9aafbee42bd8a83544ec9b97e74 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 16eac9aafbee42bd8a83544ec9b97e74, State: Running, Role: LEADER
I20240502 08:02:23.224697 7707 consensus_queue.cc:260] T cb9b5608afa2415e905426d7400abd6c P 16eac9aafbee42bd8a83544ec9b97e74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } }
I20240502 08:02:23.225634 7707 raft_consensus.cc:3037] T cb9b5608afa2415e905426d7400abd6c P 16eac9aafbee42bd8a83544ec9b97e74 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:02:23.228605 7780 raft_consensus.cc:2454] T cb9b5608afa2415e905426d7400abd6c P 36331454fb984af5a6d4f35efe0a2b00 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5cb06cedbcb4941afcceab604f34ea4 in term 2.
I20240502 08:02:23.229698 7596 leader_election.cc:302] T cb9b5608afa2415e905426d7400abd6c P d5cb06cedbcb4941afcceab604f34ea4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36331454fb984af5a6d4f35efe0a2b00, d5cb06cedbcb4941afcceab604f34ea4; no voters:
I20240502 08:02:23.230165 7707 raft_consensus.cc:2454] T cb9b5608afa2415e905426d7400abd6c P 16eac9aafbee42bd8a83544ec9b97e74 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5cb06cedbcb4941afcceab604f34ea4 in term 2.
I20240502 08:02:23.230620 7830 raft_consensus.cc:2781] T cb9b5608afa2415e905426d7400abd6c P d5cb06cedbcb4941afcceab604f34ea4 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:02:23.231488 7830 raft_consensus.cc:686] T cb9b5608afa2415e905426d7400abd6c P d5cb06cedbcb4941afcceab604f34ea4 [term 2 LEADER]: Becoming Leader. State: Replica: d5cb06cedbcb4941afcceab604f34ea4, State: Running, Role: LEADER
I20240502 08:02:23.232201 7830 consensus_queue.cc:237] T cb9b5608afa2415e905426d7400abd6c P d5cb06cedbcb4941afcceab604f34ea4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } }
I20240502 08:02:23.239424 7529 catalog_manager.cc:5484] T cb9b5608afa2415e905426d7400abd6c P d5cb06cedbcb4941afcceab604f34ea4 reported cstate change: term changed from 1 to 2, leader changed from 16eac9aafbee42bd8a83544ec9b97e74 (127.6.204.194) to d5cb06cedbcb4941afcceab604f34ea4 (127.6.204.193). New cstate: current_term: 2 leader_uuid: "d5cb06cedbcb4941afcceab604f34ea4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } health_report { overall_health: UNKNOWN } } }
I20240502 08:02:23.649201 7707 raft_consensus.cc:1261] T cb9b5608afa2415e905426d7400abd6c P 16eac9aafbee42bd8a83544ec9b97e74 [term 2 FOLLOWER]: Refusing update from remote peer d5cb06cedbcb4941afcceab604f34ea4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:02:23.650413 7830 consensus_queue.cc:1035] T cb9b5608afa2415e905426d7400abd6c P d5cb06cedbcb4941afcceab604f34ea4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:02:23.659159 7780 raft_consensus.cc:1261] T cb9b5608afa2415e905426d7400abd6c P 36331454fb984af5a6d4f35efe0a2b00 [term 2 FOLLOWER]: Refusing update from remote peer d5cb06cedbcb4941afcceab604f34ea4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:02:23.661501 7980 consensus_queue.cc:1035] T cb9b5608afa2415e905426d7400abd6c P d5cb06cedbcb4941afcceab604f34ea4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:02:24.762552 7583 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:02:24.768105 7583 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:02:24.768502 7583 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:02:26.765715 7632 consensus_queue.cc:237] T af8197b1de49493a966b8ba1146ada68 P d5cb06cedbcb4941afcceab604f34ea4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } attrs { replace: true } } peers { permanent_uuid: "89fb59c0fd364e37b3486a7b04ed9f4c" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 36987 } attrs { promote: true } }
I20240502 08:02:26.769523 7583 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:02:26.777370 7781 raft_consensus.cc:1261] T af8197b1de49493a966b8ba1146ada68 P 36331454fb984af5a6d4f35efe0a2b00 [term 1 FOLLOWER]: Refusing update from remote peer d5cb06cedbcb4941afcceab604f34ea4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:02:26.779783 7997 consensus_queue.cc:1035] T af8197b1de49493a966b8ba1146ada68 P d5cb06cedbcb4941afcceab604f34ea4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240502 08:02:26.782091 7583 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:02:26.783321 7583 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:02:26.786911 7706 raft_consensus.cc:1261] T af8197b1de49493a966b8ba1146ada68 P 16eac9aafbee42bd8a83544ec9b97e74 [term 1 FOLLOWER]: Refusing update from remote peer d5cb06cedbcb4941afcceab604f34ea4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:02:26.788838 7993 consensus_queue.cc:1035] T af8197b1de49493a966b8ba1146ada68 P d5cb06cedbcb4941afcceab604f34ea4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240502 08:02:26.792455 7993 raft_consensus.cc:2932] T af8197b1de49493a966b8ba1146ada68 P d5cb06cedbcb4941afcceab604f34ea4 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 89fb59c0fd364e37b3486a7b04ed9f4c (127.6.204.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } attrs { replace: true } } peers { permanent_uuid: "89fb59c0fd364e37b3486a7b04ed9f4c" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 36987 } attrs { promote: true } } }
W20240502 08:02:26.794987 7597 consensus_peers.cc:485] T af8197b1de49493a966b8ba1146ada68 P d5cb06cedbcb4941afcceab604f34ea4 -> Peer 89fb59c0fd364e37b3486a7b04ed9f4c (127.6.204.196:36987): Couldn't send request to peer 89fb59c0fd364e37b3486a7b04ed9f4c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: af8197b1de49493a966b8ba1146ada68. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:02:26.794330 7781 raft_consensus.cc:2932] T af8197b1de49493a966b8ba1146ada68 P 36331454fb984af5a6d4f35efe0a2b00 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 89fb59c0fd364e37b3486a7b04ed9f4c (127.6.204.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } attrs { replace: true } } peers { permanent_uuid: "89fb59c0fd364e37b3486a7b04ed9f4c" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 36987 } attrs { promote: true } } }
I20240502 08:02:26.806620 7706 raft_consensus.cc:2932] T af8197b1de49493a966b8ba1146ada68 P 16eac9aafbee42bd8a83544ec9b97e74 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 89fb59c0fd364e37b3486a7b04ed9f4c (127.6.204.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } attrs { replace: true } } peers { permanent_uuid: "89fb59c0fd364e37b3486a7b04ed9f4c" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 36987 } attrs { promote: true } } }
I20240502 08:02:26.808147 7530 catalog_manager.cc:5484] T af8197b1de49493a966b8ba1146ada68 P 36331454fb984af5a6d4f35efe0a2b00 reported cstate change: config changed from index -1 to 2, NON_VOTER 89fb59c0fd364e37b3486a7b04ed9f4c (127.6.204.196) added. New cstate: current_term: 1 leader_uuid: "d5cb06cedbcb4941afcceab604f34ea4" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } attrs { replace: true } } peers { permanent_uuid: "89fb59c0fd364e37b3486a7b04ed9f4c" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 36987 } attrs { promote: true } } }
I20240502 08:02:26.828467 7706 consensus_queue.cc:237] T b7961f100fcd49f581e2b1bfa1f1ec05 P 16eac9aafbee42bd8a83544ec9b97e74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } attrs { replace: true } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } } peers { permanent_uuid: "89fb59c0fd364e37b3486a7b04ed9f4c" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 36987 } attrs { promote: true } }
I20240502 08:02:26.835987 7781 raft_consensus.cc:1261] T b7961f100fcd49f581e2b1bfa1f1ec05 P 36331454fb984af5a6d4f35efe0a2b00 [term 1 FOLLOWER]: Refusing update from remote peer 16eac9aafbee42bd8a83544ec9b97e74: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:02:26.838337 8009 consensus_queue.cc:1035] T b7961f100fcd49f581e2b1bfa1f1ec05 P 16eac9aafbee42bd8a83544ec9b97e74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240502 08:02:26.842372 7632 raft_consensus.cc:1261] T b7961f100fcd49f581e2b1bfa1f1ec05 P d5cb06cedbcb4941afcceab604f34ea4 [term 1 FOLLOWER]: Refusing update from remote peer 16eac9aafbee42bd8a83544ec9b97e74: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:02:26.844142 7834 consensus_queue.cc:1035] T b7961f100fcd49f581e2b1bfa1f1ec05 P 16eac9aafbee42bd8a83544ec9b97e74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20240502 08:02:26.848208 7672 consensus_peers.cc:485] T b7961f100fcd49f581e2b1bfa1f1ec05 P 16eac9aafbee42bd8a83544ec9b97e74 -> Peer 89fb59c0fd364e37b3486a7b04ed9f4c (127.6.204.196:36987): Couldn't send request to peer 89fb59c0fd364e37b3486a7b04ed9f4c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b7961f100fcd49f581e2b1bfa1f1ec05. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:02:26.853386 8009 raft_consensus.cc:2932] T b7961f100fcd49f581e2b1bfa1f1ec05 P 16eac9aafbee42bd8a83544ec9b97e74 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 89fb59c0fd364e37b3486a7b04ed9f4c (127.6.204.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } attrs { replace: true } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } } peers { permanent_uuid: "89fb59c0fd364e37b3486a7b04ed9f4c" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 36987 } attrs { promote: true } } }
I20240502 08:02:26.855401 7781 raft_consensus.cc:2932] T b7961f100fcd49f581e2b1bfa1f1ec05 P 36331454fb984af5a6d4f35efe0a2b00 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 89fb59c0fd364e37b3486a7b04ed9f4c (127.6.204.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } attrs { replace: true } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } } peers { permanent_uuid: "89fb59c0fd364e37b3486a7b04ed9f4c" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 36987 } attrs { promote: true } } }
I20240502 08:02:26.857616 7632 raft_consensus.cc:2932] T b7961f100fcd49f581e2b1bfa1f1ec05 P d5cb06cedbcb4941afcceab604f34ea4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 89fb59c0fd364e37b3486a7b04ed9f4c (127.6.204.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } attrs { replace: true } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } } peers { permanent_uuid: "89fb59c0fd364e37b3486a7b04ed9f4c" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 36987 } attrs { promote: true } } }
I20240502 08:02:26.868510 7631 consensus_queue.cc:237] T d4042d449ac74cb6bbeb7ba7f90131c4 P d5cb06cedbcb4941afcceab604f34ea4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } } peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } attrs { replace: true } } peers { permanent_uuid: "89fb59c0fd364e37b3486a7b04ed9f4c" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 36987 } attrs { promote: true } }
I20240502 08:02:26.869453 7528 catalog_manager.cc:5484] T b7961f100fcd49f581e2b1bfa1f1ec05 P 36331454fb984af5a6d4f35efe0a2b00 reported cstate change: config changed from index -1 to 2, NON_VOTER 89fb59c0fd364e37b3486a7b04ed9f4c (127.6.204.196) added. New cstate: current_term: 1 leader_uuid: "16eac9aafbee42bd8a83544ec9b97e74" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } attrs { replace: true } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } } peers { permanent_uuid: "89fb59c0fd364e37b3486a7b04ed9f4c" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 36987 } attrs { promote: true } } }
I20240502 08:02:26.877276 7706 raft_consensus.cc:1261] T d4042d449ac74cb6bbeb7ba7f90131c4 P 16eac9aafbee42bd8a83544ec9b97e74 [term 1 FOLLOWER]: Refusing update from remote peer d5cb06cedbcb4941afcceab604f34ea4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 08:02:26.877875 7597 consensus_peers.cc:485] T d4042d449ac74cb6bbeb7ba7f90131c4 P d5cb06cedbcb4941afcceab604f34ea4 -> Peer 89fb59c0fd364e37b3486a7b04ed9f4c (127.6.204.196:36987): Couldn't send request to peer 89fb59c0fd364e37b3486a7b04ed9f4c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d4042d449ac74cb6bbeb7ba7f90131c4. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:02:26.878861 7993 consensus_queue.cc:1035] T d4042d449ac74cb6bbeb7ba7f90131c4 P d5cb06cedbcb4941afcceab604f34ea4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:02:26.879483 7781 raft_consensus.cc:1261] T d4042d449ac74cb6bbeb7ba7f90131c4 P 36331454fb984af5a6d4f35efe0a2b00 [term 1 FOLLOWER]: Refusing update from remote peer d5cb06cedbcb4941afcceab604f34ea4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:02:26.880880 7997 consensus_queue.cc:1035] T d4042d449ac74cb6bbeb7ba7f90131c4 P d5cb06cedbcb4941afcceab604f34ea4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:02:26.887810 7997 raft_consensus.cc:2932] T d4042d449ac74cb6bbeb7ba7f90131c4 P d5cb06cedbcb4941afcceab604f34ea4 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 89fb59c0fd364e37b3486a7b04ed9f4c (127.6.204.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } } peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } attrs { replace: true } } peers { permanent_uuid: "89fb59c0fd364e37b3486a7b04ed9f4c" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 36987 } attrs { promote: true } } }
I20240502 08:02:26.889556 7706 raft_consensus.cc:2932] T d4042d449ac74cb6bbeb7ba7f90131c4 P 16eac9aafbee42bd8a83544ec9b97e74 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 89fb59c0fd364e37b3486a7b04ed9f4c (127.6.204.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } } peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } attrs { replace: true } } peers { permanent_uuid: "89fb59c0fd364e37b3486a7b04ed9f4c" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 36987 } attrs { promote: true } } }
I20240502 08:02:26.906579 7528 catalog_manager.cc:5484] T d4042d449ac74cb6bbeb7ba7f90131c4 P 16eac9aafbee42bd8a83544ec9b97e74 reported cstate change: config changed from index -1 to 2, NON_VOTER 89fb59c0fd364e37b3486a7b04ed9f4c (127.6.204.196) added. New cstate: current_term: 1 leader_uuid: "d5cb06cedbcb4941afcceab604f34ea4" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } } peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } attrs { replace: true } } peers { permanent_uuid: "89fb59c0fd364e37b3486a7b04ed9f4c" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 36987 } attrs { promote: true } } }
I20240502 08:02:26.924727 7781 raft_consensus.cc:2932] T d4042d449ac74cb6bbeb7ba7f90131c4 P 36331454fb984af5a6d4f35efe0a2b00 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 89fb59c0fd364e37b3486a7b04ed9f4c (127.6.204.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } } peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } attrs { replace: true } } peers { permanent_uuid: "89fb59c0fd364e37b3486a7b04ed9f4c" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 36987 } attrs { promote: true } } }
I20240502 08:02:26.931922 7780 consensus_queue.cc:237] T f8a4589c0a9b417188a2d884011b0240 P 36331454fb984af5a6d4f35efe0a2b00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } attrs { replace: true } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } } peers { permanent_uuid: "89fb59c0fd364e37b3486a7b04ed9f4c" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 36987 } attrs { promote: true } }
I20240502 08:02:26.943356 7706 raft_consensus.cc:1261] T f8a4589c0a9b417188a2d884011b0240 P 16eac9aafbee42bd8a83544ec9b97e74 [term 1 FOLLOWER]: Refusing update from remote peer 36331454fb984af5a6d4f35efe0a2b00: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:02:26.944839 7833 consensus_queue.cc:1035] T f8a4589c0a9b417188a2d884011b0240 P 36331454fb984af5a6d4f35efe0a2b00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240502 08:02:26.948637 7631 raft_consensus.cc:1261] T f8a4589c0a9b417188a2d884011b0240 P d5cb06cedbcb4941afcceab604f34ea4 [term 1 FOLLOWER]: Refusing update from remote peer 36331454fb984af5a6d4f35efe0a2b00: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:02:26.950017 8019 consensus_queue.cc:1035] T f8a4589c0a9b417188a2d884011b0240 P 36331454fb984af5a6d4f35efe0a2b00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 08:02:26.952116 7746 consensus_peers.cc:485] T f8a4589c0a9b417188a2d884011b0240 P 36331454fb984af5a6d4f35efe0a2b00 -> Peer 89fb59c0fd364e37b3486a7b04ed9f4c (127.6.204.196:36987): Couldn't send request to peer 89fb59c0fd364e37b3486a7b04ed9f4c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f8a4589c0a9b417188a2d884011b0240. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:02:26.958240 7833 raft_consensus.cc:2932] T f8a4589c0a9b417188a2d884011b0240 P 36331454fb984af5a6d4f35efe0a2b00 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 89fb59c0fd364e37b3486a7b04ed9f4c (127.6.204.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } attrs { replace: true } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } } peers { permanent_uuid: "89fb59c0fd364e37b3486a7b04ed9f4c" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 36987 } attrs { promote: true } } }
I20240502 08:02:26.959779 7631 raft_consensus.cc:2932] T f8a4589c0a9b417188a2d884011b0240 P d5cb06cedbcb4941afcceab604f34ea4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 89fb59c0fd364e37b3486a7b04ed9f4c (127.6.204.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } attrs { replace: true } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } } peers { permanent_uuid: "89fb59c0fd364e37b3486a7b04ed9f4c" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 36987 } attrs { promote: true } } }
I20240502 08:02:26.967190 7706 raft_consensus.cc:2932] T f8a4589c0a9b417188a2d884011b0240 P 16eac9aafbee42bd8a83544ec9b97e74 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 89fb59c0fd364e37b3486a7b04ed9f4c (127.6.204.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } attrs { replace: true } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } } peers { permanent_uuid: "89fb59c0fd364e37b3486a7b04ed9f4c" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 36987 } attrs { promote: true } } }
I20240502 08:02:26.972118 7530 catalog_manager.cc:5484] T f8a4589c0a9b417188a2d884011b0240 P d5cb06cedbcb4941afcceab604f34ea4 reported cstate change: config changed from index -1 to 2, NON_VOTER 89fb59c0fd364e37b3486a7b04ed9f4c (127.6.204.196) added. New cstate: current_term: 1 leader_uuid: "36331454fb984af5a6d4f35efe0a2b00" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d5cb06cedbcb4941afcceab604f34ea4" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42525 } } peers { permanent_uuid: "16eac9aafbee42bd8a83544ec9b97e74" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 38399 } attrs { replace: true } } peers { permanent_uuid: "36331454fb984af5a6d4f35efe0a2b00" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40297 } } peers { permanent_uuid: "89fb59c0fd364e37b3486a7b04ed9f4c" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 36987 } attrs { promote: true } } }
I20240502 08:02:27.172103 6963 tablet_server.cc:178] TabletServer@127.6.204.193:0 shutting down...
W20240502 08:02:27.184501 7514 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 5 similar messages]
W20240502 08:02:27.188726 7578 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240502 08:02:27.189043 7578 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240502 08:02:27.191627 8025 ts_tablet_manager.cc:902] T af8197b1de49493a966b8ba1146ada68 P 89fb59c0fd364e37b3486a7b04ed9f4c: Initiating tablet copy from peer d5cb06cedbcb4941afcceab604f34ea4 (127.6.204.193:42525)
W20240502 08:02:27.193949 7578 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240502 08:02:27.194398 7578 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240502 08:02:27.202045 8025 tablet_copy_client.cc:308] T af8197b1de49493a966b8ba1146ada68 P 89fb59c0fd364e37b3486a7b04ed9f4c: tablet copy: Beginning tablet copy session from remote peer at address 127.6.204.193:42525
I20240502 08:02:27.221341 6963 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 08:02:27.222651 6963 tablet_replica.cc:330] T ac59612a1fa642f6b7dfd4d899bd6f9d P d5cb06cedbcb4941afcceab604f34ea4: stopping tablet replica
I20240502 08:02:27.223552 6963 raft_consensus.cc:2229] T ac59612a1fa642f6b7dfd4d899bd6f9d P d5cb06cedbcb4941afcceab604f34ea4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:02:27.224182 6963 raft_consensus.cc:2258] T ac59612a1fa642f6b7dfd4d899bd6f9d P d5cb06cedbcb4941afcceab604f34ea4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:02:27.227008 6963 tablet_replica.cc:330] T cb9b5608afa2415e905426d7400abd6c P d5cb06cedbcb4941afcceab604f34ea4: stopping tablet replica
I20240502 08:02:27.227689 6963 raft_consensus.cc:2229] T cb9b5608afa2415e905426d7400abd6c P d5cb06cedbcb4941afcceab604f34ea4 [term 2 LEADER]: Raft consensus shutting down.
I20240502 08:02:27.228677 6963 raft_consensus.cc:2258] T cb9b5608afa2415e905426d7400abd6c P d5cb06cedbcb4941afcceab604f34ea4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:02:27.230444 6963 tablet_replica.cc:330] T 8e0f7109a1a94eb999a919ed98dea1ce P d5cb06cedbcb4941afcceab604f34ea4: stopping tablet replica
I20240502 08:02:27.231151 6963 raft_consensus.cc:2229] T 8e0f7109a1a94eb999a919ed98dea1ce P d5cb06cedbcb4941afcceab604f34ea4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:02:27.231549 6963 raft_consensus.cc:2258] T 8e0f7109a1a94eb999a919ed98dea1ce P d5cb06cedbcb4941afcceab604f34ea4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:02:27.233399 6963 tablet_replica.cc:330] T 8f45c7e5e5cf4b4a84127573dd446bde P d5cb06cedbcb4941afcceab604f34ea4: stopping tablet replica
I20240502 08:02:27.234056 6963 raft_consensus.cc:2229] T 8f45c7e5e5cf4b4a84127573dd446bde P d5cb06cedbcb4941afcceab604f34ea4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:02:27.234539 6963 raft_consensus.cc:2258] T 8f45c7e5e5cf4b4a84127573dd446bde P d5cb06cedbcb4941afcceab604f34ea4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:02:27.236534 6963 tablet_replica.cc:330] T af8197b1de49493a966b8ba1146ada68 P d5cb06cedbcb4941afcceab604f34ea4: stopping tablet replica
I20240502 08:02:27.237017 6963 raft_consensus.cc:2229] T af8197b1de49493a966b8ba1146ada68 P d5cb06cedbcb4941afcceab604f34ea4 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:02:27.237934 6963 raft_consensus.cc:2258] T af8197b1de49493a966b8ba1146ada68 P d5cb06cedbcb4941afcceab604f34ea4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:02:27.240733 6963 tablet_replica.cc:330] T f8a4589c0a9b417188a2d884011b0240 P d5cb06cedbcb4941afcceab604f34ea4: stopping tablet replica
I20240502 08:02:27.241405 6963 raft_consensus.cc:2229] T f8a4589c0a9b417188a2d884011b0240 P d5cb06cedbcb4941afcceab604f34ea4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:02:27.241926 6963 raft_consensus.cc:2258] T f8a4589c0a9b417188a2d884011b0240 P d5cb06cedbcb4941afcceab604f34ea4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:02:27.244310 6963 tablet_replica.cc:330] T b7961f100fcd49f581e2b1bfa1f1ec05 P d5cb06cedbcb4941afcceab604f34ea4: stopping tablet replica
I20240502 08:02:27.244779 6963 raft_consensus.cc:2229] T b7961f100fcd49f581e2b1bfa1f1ec05 P d5cb06cedbcb4941afcceab604f34ea4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:02:27.245210 6963 raft_consensus.cc:2258] T b7961f100fcd49f581e2b1bfa1f1ec05 P d5cb06cedbcb4941afcceab604f34ea4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:02:27.247440 6963 tablet_replica.cc:330] T d4042d449ac74cb6bbeb7ba7f90131c4 P d5cb06cedbcb4941afcceab604f34ea4: stopping tablet replica
I20240502 08:02:27.247924 6963 raft_consensus.cc:2229] T d4042d449ac74cb6bbeb7ba7f90131c4 P d5cb06cedbcb4941afcceab604f34ea4 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:02:27.248997 6963 raft_consensus.cc:2258] T d4042d449ac74cb6bbeb7ba7f90131c4 P d5cb06cedbcb4941afcceab604f34ea4 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240502 08:02:27.274124 7744 consensus_peers.cc:485] T 8f45c7e5e5cf4b4a84127573dd446bde P 36331454fb984af5a6d4f35efe0a2b00 -> Peer d5cb06cedbcb4941afcceab604f34ea4 (127.6.204.193:42525): Couldn't send request to peer d5cb06cedbcb4941afcceab604f34ea4. Status: Network error: Client connection negotiation failed: client connection to 127.6.204.193:42525: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:02:27.281791 6963 tablet_server.cc:194] TabletServer@127.6.204.193:0 shutdown complete.
W20240502 08:02:27.295465 7670 consensus_peers.cc:485] T 8e0f7109a1a94eb999a919ed98dea1ce P 16eac9aafbee42bd8a83544ec9b97e74 -> Peer d5cb06cedbcb4941afcceab604f34ea4 (127.6.204.193:42525): Couldn't send request to peer d5cb06cedbcb4941afcceab604f34ea4. Status: Network error: Client connection negotiation failed: client connection to 127.6.204.193:42525: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:02:27.304800 6963 tablet_server.cc:178] TabletServer@127.6.204.194:0 shutting down...
W20240502 08:02:27.311827 7578 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240502 08:02:27.312101 7578 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240502 08:02:27.330826 7897 connection.cc:441] server connection from 127.0.0.1:47026 torn down before Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:47026 (request call id 2) could send its response
I20240502 08:02:27.338073 6963 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 08:02:27.339025 6963 tablet_replica.cc:330] T 8f45c7e5e5cf4b4a84127573dd446bde P 16eac9aafbee42bd8a83544ec9b97e74: stopping tablet replica
I20240502 08:02:27.339777 6963 raft_consensus.cc:2229] T 8f45c7e5e5cf4b4a84127573dd446bde P 16eac9aafbee42bd8a83544ec9b97e74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:02:27.340276 6963 raft_consensus.cc:2258] T 8f45c7e5e5cf4b4a84127573dd446bde P 16eac9aafbee42bd8a83544ec9b97e74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:02:27.342727 6963 tablet_replica.cc:330] T af8197b1de49493a966b8ba1146ada68 P 16eac9aafbee42bd8a83544ec9b97e74: stopping tablet replica
I20240502 08:02:27.343307 6963 raft_consensus.cc:2229] T af8197b1de49493a966b8ba1146ada68 P 16eac9aafbee42bd8a83544ec9b97e74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:02:27.343863 6963 raft_consensus.cc:2258] T af8197b1de49493a966b8ba1146ada68 P 16eac9aafbee42bd8a83544ec9b97e74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:02:27.345872 8025 ts_tablet_manager.cc:902] T b7961f100fcd49f581e2b1bfa1f1ec05 P 89fb59c0fd364e37b3486a7b04ed9f4c: Initiating tablet copy from peer 16eac9aafbee42bd8a83544ec9b97e74 (127.6.204.194:38399)
I20240502 08:02:27.346442 6963 tablet_replica.cc:330] T b7961f100fcd49f581e2b1bfa1f1ec05 P 16eac9aafbee42bd8a83544ec9b97e74: stopping tablet replica
I20240502 08:02:27.347137 6963 raft_consensus.cc:2229] T b7961f100fcd49f581e2b1bfa1f1ec05 P 16eac9aafbee42bd8a83544ec9b97e74 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:02:27.348018 6963 raft_consensus.cc:2258] T b7961f100fcd49f581e2b1bfa1f1ec05 P 16eac9aafbee42bd8a83544ec9b97e74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:02:27.350454 6963 tablet_replica.cc:330] T f8a4589c0a9b417188a2d884011b0240 P 16eac9aafbee42bd8a83544ec9b97e74: stopping tablet replica
I20240502 08:02:27.351112 6963 raft_consensus.cc:2229] T f8a4589c0a9b417188a2d884011b0240 P 16eac9aafbee42bd8a83544ec9b97e74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:02:27.351589 6963 raft_consensus.cc:2258] T f8a4589c0a9b417188a2d884011b0240 P 16eac9aafbee42bd8a83544ec9b97e74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:02:27.353816 6963 tablet_replica.cc:330] T d4042d449ac74cb6bbeb7ba7f90131c4 P 16eac9aafbee42bd8a83544ec9b97e74: stopping tablet replica
I20240502 08:02:27.354409 6963 raft_consensus.cc:2229] T d4042d449ac74cb6bbeb7ba7f90131c4 P 16eac9aafbee42bd8a83544ec9b97e74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:02:27.354900 6963 raft_consensus.cc:2258] T d4042d449ac74cb6bbeb7ba7f90131c4 P 16eac9aafbee42bd8a83544ec9b97e74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:02:27.357756 6963 tablet_replica.cc:330] T 8e0f7109a1a94eb999a919ed98dea1ce P 16eac9aafbee42bd8a83544ec9b97e74: stopping tablet replica
I20240502 08:02:27.358340 6963 raft_consensus.cc:2229] T 8e0f7109a1a94eb999a919ed98dea1ce P 16eac9aafbee42bd8a83544ec9b97e74 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:02:27.359361 6963 raft_consensus.cc:2258] T 8e0f7109a1a94eb999a919ed98dea1ce P 16eac9aafbee42bd8a83544ec9b97e74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:02:27.361333 6963 tablet_replica.cc:330] T cb9b5608afa2415e905426d7400abd6c P 16eac9aafbee42bd8a83544ec9b97e74: stopping tablet replica
I20240502 08:02:27.361464 8025 tablet_copy_client.cc:308] T b7961f100fcd49f581e2b1bfa1f1ec05 P 89fb59c0fd364e37b3486a7b04ed9f4c: tablet copy: Beginning tablet copy session from remote peer at address 127.6.204.194:38399
I20240502 08:02:27.362003 6963 raft_consensus.cc:2229] T cb9b5608afa2415e905426d7400abd6c P 16eac9aafbee42bd8a83544ec9b97e74 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:02:27.362545 6963 raft_consensus.cc:2258] T cb9b5608afa2415e905426d7400abd6c P 16eac9aafbee42bd8a83544ec9b97e74 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:02:27.364826 6963 tablet_replica.cc:330] T ac59612a1fa642f6b7dfd4d899bd6f9d P 16eac9aafbee42bd8a83544ec9b97e74: stopping tablet replica
I20240502 08:02:27.365446 6963 raft_consensus.cc:2229] T ac59612a1fa642f6b7dfd4d899bd6f9d P 16eac9aafbee42bd8a83544ec9b97e74 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:02:27.366302 6963 raft_consensus.cc:2258] T ac59612a1fa642f6b7dfd4d899bd6f9d P 16eac9aafbee42bd8a83544ec9b97e74 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240502 08:02:27.371352 7746 consensus_peers.cc:485] T 8f45c7e5e5cf4b4a84127573dd446bde P 36331454fb984af5a6d4f35efe0a2b00 -> Peer 16eac9aafbee42bd8a83544ec9b97e74 (127.6.204.194:38399): Couldn't send request to peer 16eac9aafbee42bd8a83544ec9b97e74. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240502 08:02:27.388554 7897 connection.cc:441] server connection from 127.0.0.1:47032 torn down before Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:47032 (request call id 1) could send its response
I20240502 08:02:27.390290 6963 tablet_server.cc:194] TabletServer@127.6.204.194:0 shutdown complete.
I20240502 08:02:27.411118 6963 tablet_server.cc:178] TabletServer@127.6.204.195:0 shutting down...
W20240502 08:02:27.416652 7578 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240502 08:02:27.417915 7578 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240502 08:02:27.434862 8025 ts_tablet_manager.cc:902] T f8a4589c0a9b417188a2d884011b0240 P 89fb59c0fd364e37b3486a7b04ed9f4c: Initiating tablet copy from peer 36331454fb984af5a6d4f35efe0a2b00 (127.6.204.195:40297)
I20240502 08:02:27.436923 6963 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 08:02:27.438056 6963 tablet_replica.cc:330] T 8f45c7e5e5cf4b4a84127573dd446bde P 36331454fb984af5a6d4f35efe0a2b00: stopping tablet replica
I20240502 08:02:27.439141 6963 raft_consensus.cc:2229] T 8f45c7e5e5cf4b4a84127573dd446bde P 36331454fb984af5a6d4f35efe0a2b00 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:02:27.440471 6963 raft_consensus.cc:2258] T 8f45c7e5e5cf4b4a84127573dd446bde P 36331454fb984af5a6d4f35efe0a2b00 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240502 08:02:27.443857 7744 consensus_peers.cc:485] T f8a4589c0a9b417188a2d884011b0240 P 36331454fb984af5a6d4f35efe0a2b00 -> Peer d5cb06cedbcb4941afcceab604f34ea4 (127.6.204.193:42525): Couldn't send request to peer d5cb06cedbcb4941afcceab604f34ea4. Status: Network error: Client connection negotiation failed: client connection to 127.6.204.193:42525: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:02:27.444041 6963 tablet_replica.cc:330] T af8197b1de49493a966b8ba1146ada68 P 36331454fb984af5a6d4f35efe0a2b00: stopping tablet replica
I20240502 08:02:27.445147 6963 raft_consensus.cc:2229] T af8197b1de49493a966b8ba1146ada68 P 36331454fb984af5a6d4f35efe0a2b00 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:02:27.445945 6963 raft_consensus.cc:2258] T af8197b1de49493a966b8ba1146ada68 P 36331454fb984af5a6d4f35efe0a2b00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:02:27.448640 8025 tablet_copy_client.cc:308] T f8a4589c0a9b417188a2d884011b0240 P 89fb59c0fd364e37b3486a7b04ed9f4c: tablet copy: Beginning tablet copy session from remote peer at address 127.6.204.195:40297
I20240502 08:02:27.449371 6963 tablet_replica.cc:330] T ac59612a1fa642f6b7dfd4d899bd6f9d P 36331454fb984af5a6d4f35efe0a2b00: stopping tablet replica
I20240502 08:02:27.450089 6963 raft_consensus.cc:2229] T ac59612a1fa642f6b7dfd4d899bd6f9d P 36331454fb984af5a6d4f35efe0a2b00 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:02:27.450634 6963 raft_consensus.cc:2258] T ac59612a1fa642f6b7dfd4d899bd6f9d P 36331454fb984af5a6d4f35efe0a2b00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:02:27.452827 6963 tablet_replica.cc:330] T 8e0f7109a1a94eb999a919ed98dea1ce P 36331454fb984af5a6d4f35efe0a2b00: stopping tablet replica
I20240502 08:02:27.453394 6963 raft_consensus.cc:2229] T 8e0f7109a1a94eb999a919ed98dea1ce P 36331454fb984af5a6d4f35efe0a2b00 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:02:27.453770 6963 raft_consensus.cc:2258] T 8e0f7109a1a94eb999a919ed98dea1ce P 36331454fb984af5a6d4f35efe0a2b00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:02:27.456156 6963 tablet_replica.cc:330] T cb9b5608afa2415e905426d7400abd6c P 36331454fb984af5a6d4f35efe0a2b00: stopping tablet replica
I20240502 08:02:27.456859 6963 raft_consensus.cc:2229] T cb9b5608afa2415e905426d7400abd6c P 36331454fb984af5a6d4f35efe0a2b00 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:02:27.457288 6963 raft_consensus.cc:2258] T cb9b5608afa2415e905426d7400abd6c P 36331454fb984af5a6d4f35efe0a2b00 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:02:27.458916 6963 tablet_replica.cc:330] T f8a4589c0a9b417188a2d884011b0240 P 36331454fb984af5a6d4f35efe0a2b00: stopping tablet replica
I20240502 08:02:27.459369 6963 raft_consensus.cc:2229] T f8a4589c0a9b417188a2d884011b0240 P 36331454fb984af5a6d4f35efe0a2b00 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:02:27.460103 6963 raft_consensus.cc:2258] T f8a4589c0a9b417188a2d884011b0240 P 36331454fb984af5a6d4f35efe0a2b00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:02:27.462184 6963 tablet_replica.cc:330] T b7961f100fcd49f581e2b1bfa1f1ec05 P 36331454fb984af5a6d4f35efe0a2b00: stopping tablet replica
I20240502 08:02:27.462685 6963 raft_consensus.cc:2229] T b7961f100fcd49f581e2b1bfa1f1ec05 P 36331454fb984af5a6d4f35efe0a2b00 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:02:27.463114 6963 raft_consensus.cc:2258] T b7961f100fcd49f581e2b1bfa1f1ec05 P 36331454fb984af5a6d4f35efe0a2b00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:02:27.465235 6963 tablet_replica.cc:330] T d4042d449ac74cb6bbeb7ba7f90131c4 P 36331454fb984af5a6d4f35efe0a2b00: stopping tablet replica
I20240502 08:02:27.465667 6963 raft_consensus.cc:2229] T d4042d449ac74cb6bbeb7ba7f90131c4 P 36331454fb984af5a6d4f35efe0a2b00 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:02:27.466063 6963 raft_consensus.cc:2258] T d4042d449ac74cb6bbeb7ba7f90131c4 P 36331454fb984af5a6d4f35efe0a2b00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:02:27.494493 6963 tablet_server.cc:194] TabletServer@127.6.204.195:0 shutdown complete.
W20240502 08:02:27.509879 7892 connection.cc:441] server connection from 127.0.0.1:47042 torn down before Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:47042 (request call id 1) could send its response
I20240502 08:02:27.512442 6963 tablet_server.cc:178] TabletServer@127.6.204.196:0 shutting down...
I20240502 08:02:27.531582 6963 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 08:02:27.548437 6963 tablet_server.cc:194] TabletServer@127.6.204.196:0 shutdown complete.
I20240502 08:02:27.556738 6963 master.cc:537] Master@127.6.204.254:40853 shutting down...
I20240502 08:02:27.575834 6963 raft_consensus.cc:2229] T 00000000000000000000000000000000 P d63c312fa1b34251997988782a79379c [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:02:27.576375 6963 raft_consensus.cc:2258] T 00000000000000000000000000000000 P d63c312fa1b34251997988782a79379c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:02:27.576678 6963 tablet_replica.cc:330] T 00000000000000000000000000000000 P d63c312fa1b34251997988782a79379c: stopping tablet replica
I20240502 08:02:27.594334 6963 master.cc:552] Master@127.6.204.254:40853 shutdown complete.
[ OK ] AutoRebalancerTest.AutoRebalancingTurnOffAndOn (9069 ms)
[ RUN ] AutoRebalancerTest.NextLeaderResumesAutoRebalancing
I20240502 08:02:27.633394 6963 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.6.204.254:33115,127.6.204.253:42483,127.6.204.252:46535
I20240502 08:02:27.634812 6963 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:02:27.640317 8044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:02:27.640974 8045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:02:27.641662 8047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:02:27.642300 6963 server_base.cc:1034] running on GCE node
I20240502 08:02:27.643190 6963 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:02:27.643366 6963 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:02:27.643501 6963 hybrid_clock.cc:648] HybridClock initialized: now 1714636947643488 us; error 0 us; skew 500 ppm
I20240502 08:02:27.643932 6963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:02:27.646405 6963 webserver.cc:460] Webserver started at http://127.6.204.254:34499/ using document root <none> and password file <none>
I20240502 08:02:27.646852 6963 fs_manager.cc:359] Metadata directory not provided
I20240502 08:02:27.647010 6963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:02:27.647230 6963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:02:27.648311 6963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714636933625983-6963-0/minicluster-data/master-0-root/instance:
uuid: "e80f89dd462b42ad8b210a98f0891253"
format_stamp: "Formatted at 2024-05-02 08:02:27 on dist-test-slave-32xs"
I20240502 08:02:27.652501 6963 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240502 08:02:27.655870 8052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:02:27.656558 6963 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 08:02:27.656836 6963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714636933625983-6963-0/minicluster-data/master-0-root
uuid: "e80f89dd462b42ad8b210a98f0891253"
format_stamp: "Formatted at 2024-05-02 08:02:27 on dist-test-slave-32xs"
I20240502 08:02:27.657092 6963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714636933625983-6963-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714636933625983-6963-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714636933625983-6963-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:02:27.677491 6963 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:02:27.678757 6963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:02:27.715420 6963 rpc_server.cc:307] RPC server started. Bound to: 127.6.204.254:33115
I20240502 08:02:27.715512 8103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.204.254:33115 every 8 connection(s)
I20240502 08:02:27.719249 8104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:02:27.719370 6963 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:02:27.724486 8106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:02:27.725422 8107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:02:27.727108 8104 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.204.254" port: 33115 } has no permanent_uuid. Determining permanent_uuid...
I20240502 08:02:27.728415 6963 server_base.cc:1034] running on GCE node
W20240502 08:02:27.728523 8110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:02:27.729472 6963 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:02:27.729683 6963 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:02:27.729853 6963 hybrid_clock.cc:648] HybridClock initialized: now 1714636947729835 us; error 0 us; skew 500 ppm
I20240502 08:02:27.730367 6963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:02:27.733191 6963 webserver.cc:460] Webserver started at http://127.6.204.253:38133/ using document root <none> and password file <none>
I20240502 08:02:27.733626 6963 fs_manager.cc:359] Metadata directory not provided
I20240502 08:02:27.733809 6963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:02:27.734050 6963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:02:27.735055 6963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714636933625983-6963-0/minicluster-data/master-1-root/instance:
uuid: "1f7d11296f7f4716b9eb5602aa3c842d"
format_stamp: "Formatted at 2024-05-02 08:02:27 on dist-test-slave-32xs"
I20240502 08:02:27.739990 6963 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 08:02:27.740312 8104 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.204.253" port: 42483 } has no permanent_uuid. Determining permanent_uuid...
I20240502 08:02:27.744035 8116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:02:27.744817 6963 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20240502 08:02:27.745167 6963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714636933625983-6963-0/minicluster-data/master-1-root
uuid: "1f7d11296f7f4716b9eb5602aa3c842d"
format_stamp: "Formatted at 2024-05-02 08:02:27 on dist-test-slave-32xs"
I20240502 08:02:27.745558 6963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714636933625983-6963-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714636933625983-6963-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714636933625983-6963-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20240502 08:02:27.745997 8104 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.6.204.253:42483: Network error: Client connection negotiation failed: client connection to 127.6.204.253:42483: connect: Connection refused (error 111)
I20240502 08:02:27.755270 6963 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:02:27.756350 6963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:02:27.792649 6963 rpc_server.cc:307] RPC server started. Bound to: 127.6.204.253:42483
I20240502 08:02:27.792774 8168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.204.253:42483 every 8 connection(s)
I20240502 08:02:27.796300 8169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:02:27.796496 6963 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 08:02:27.798517 8104 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.204.253" port: 42483 } attempt: 1
W20240502 08:02:27.801944 8171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:02:27.804889 8169 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.204.254" port: 33115 } has no permanent_uuid. Determining permanent_uuid...
W20240502 08:02:27.807065 8172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:02:27.820274 6963 server_base.cc:1034] running on GCE node
W20240502 08:02:27.821250 8176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:02:27.821542 8104 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.204.252" port: 46535 } has no permanent_uuid. Determining permanent_uuid...
I20240502 08:02:27.822365 6963 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:02:27.823397 6963 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:02:27.823668 6963 hybrid_clock.cc:648] HybridClock initialized: now 1714636947823649 us; error 0 us; skew 500 ppm
I20240502 08:02:27.823907 8169 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.204.253" port: 42483 } has no permanent_uuid. Determining permanent_uuid...
I20240502 08:02:27.824249 6963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
W20240502 08:02:27.826565 8104 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.6.204.252:46535: Network error: Client connection negotiation failed: client connection to 127.6.204.252:46535: connect: Connection refused (error 111)
I20240502 08:02:27.828555 6963 webserver.cc:460] Webserver started at http://127.6.204.252:43475/ using document root <none> and password file <none>
I20240502 08:02:27.829044 6963 fs_manager.cc:359] Metadata directory not provided
I20240502 08:02:27.829221 6963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:02:27.829453 6963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:02:27.830534 6963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714636933625983-6963-0/minicluster-data/master-2-root/instance:
uuid: "55808e5996304e98b57f3315454e4368"
format_stamp: "Formatted at 2024-05-02 08:02:27 on dist-test-slave-32xs"
I20240502 08:02:27.834721 8169 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.204.252" port: 46535 } has no permanent_uuid. Determining permanent_uuid...
I20240502 08:02:27.835722 6963 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
W20240502 08:02:27.839964 8169 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.6.204.252:46535: Network error: Client connection negotiation failed: client connection to 127.6.204.252:46535: connect: Connection refused (error 111)
I20240502 08:02:27.840257 8182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:02:27.841132 6963 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240502 08:02:27.841390 6963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714636933625983-6963-0/minicluster-data/master-2-root
uuid: "55808e5996304e98b57f3315454e4368"
format_stamp: "Formatted at 2024-05-02 08:02:27 on dist-test-slave-32xs"
I20240502 08:02:27.841637 6963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714636933625983-6963-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714636933625983-6963-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714636933625983-6963-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:02:27.845083 8104 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.204.252" port: 46535 } attempt: 1
W20240502 08:02:27.849282 8104 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.6.204.252:46535: Network error: Client connection negotiation failed: client connection to 127.6.204.252:46535: connect: Connection refused (error 111)
I20240502 08:02:27.851852 6963 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:02:27.853081 6963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:02:27.889423 8169 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.204.252" port: 46535 } attempt: 1
I20240502 08:02:27.893327 6963 rpc_server.cc:307] RPC server started. Bound to: 127.6.204.252:46535
I20240502 08:02:27.893441 8233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.204.252:46535 every 8 connection(s)
I20240502 08:02:27.896821 6963 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 08:02:27.898002 8235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:02:27.905426 8235 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.204.254" port: 33115 } has no permanent_uuid. Determining permanent_uuid...
I20240502 08:02:27.911758 8169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f7d11296f7f4716b9eb5602aa3c842d: Bootstrap starting.
I20240502 08:02:27.916777 8104 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.204.252" port: 46535 } attempt: 2
I20240502 08:02:27.917593 8169 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f7d11296f7f4716b9eb5602aa3c842d: Neither blocks nor log segments found. Creating new log.
I20240502 08:02:27.917908 8235 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.204.253" port: 42483 } has no permanent_uuid. Determining permanent_uuid...
I20240502 08:02:27.924626 8169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f7d11296f7f4716b9eb5602aa3c842d: No bootstrap required, opened a new log
I20240502 08:02:27.927507 8169 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f7d11296f7f4716b9eb5602aa3c842d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e80f89dd462b42ad8b210a98f0891253" member_type: VOTER last_known_addr { host: "127.6.204.254" port: 33115 } } peers { permanent_uuid: "1f7d11296f7f4716b9eb5602aa3c842d" member_type: VOTER last_known_addr { host: "127.6.204.253" port: 42483 } } peers { permanent_uuid: "55808e5996304e98b57f3315454e4368" member_type: VOTER last_known_addr { host: "127.6.204.252" port: 46535 } }
I20240502 08:02:27.928187 8169 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1f7d11296f7f4716b9eb5602aa3c842d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:02:27.928484 8169 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f7d11296f7f4716b9eb5602aa3c842d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f7d11296f7f4716b9eb5602aa3c842d, State: Initialized, Role: FOLLOWER
I20240502 08:02:27.929531 8235 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.204.252" port: 46535 } has no permanent_uuid. Determining permanent_uuid...
I20240502 08:02:27.929174 8169 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f7d11296f7f4716b9eb5602aa3c842d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e80f89dd462b42ad8b210a98f0891253" member_type: VOTER last_known_addr { host: "127.6.204.254" port: 33115 } } peers { permanent_uuid: "1f7d11296f7f4716b9eb5602aa3c842d" member_type: VOTER last_known_addr { host: "127.6.204.253" port: 42483 } } peers { permanent_uuid: "55808e5996304e98b57f3315454e4368" member_type: VOTER last_known_addr { host: "127.6.204.252" port: 46535 } }
I20240502 08:02:27.932678 8239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f7d11296f7f4716b9eb5602aa3c842d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e80f89dd462b42ad8b210a98f0891253" member_type: VOTER last_known_addr { host: "127.6.204.254" port: 33115 } } peers { permanent_uuid: "1f7d11296f7f4716b9eb5602aa3c842d" member_type: VOTER last_known_addr { host: "127.6.204.253" port: 42483 } } peers { permanent_uuid: "55808e5996304e98b57f3315454e4368" member_type: VOTER last_known_addr { host: "127.6.204.252" port: 46535 } } }
I20240502 08:02:27.933310 8169 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f7d11296f7f4716b9eb5602aa3c842d [sys.catalog]: configured and running, proceeding with master startup.
I20240502 08:02:27.933498 8239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f7d11296f7f4716b9eb5602aa3c842d [sys.catalog]: This master's current role is: FOLLOWER
I20240502 08:02:27.934864 8104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e80f89dd462b42ad8b210a98f0891253: Bootstrap starting.
I20240502 08:02:27.941165 8104 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e80f89dd462b42ad8b210a98f0891253: Neither blocks nor log segments found. Creating new log.
I20240502 08:02:27.952459 8104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e80f89dd462b42ad8b210a98f0891253: No bootstrap required, opened a new log
I20240502 08:02:27.953871 8235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55808e5996304e98b57f3315454e4368: Bootstrap starting.
I20240502 08:02:27.954538 8104 raft_consensus.cc:348] T 00000000000000000000000000000000 P e80f89dd462b42ad8b210a98f0891253 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e80f89dd462b42ad8b210a98f0891253" member_type: VOTER last_known_addr { host: "127.6.204.254" port: 33115 } } peers { permanent_uuid: "1f7d11296f7f4716b9eb5602aa3c842d" member_type: VOTER last_known_addr { host: "127.6.204.253" port: 42483 } } peers { permanent_uuid: "55808e5996304e98b57f3315454e4368" member_type: VOTER last_known_addr { host: "127.6.204.252" port: 46535 } }
I20240502 08:02:27.955497 8104 raft_consensus.cc:374] T 00000000000000000000000000000000 P e80f89dd462b42ad8b210a98f0891253 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:02:27.955886 8104 raft_consensus.cc:729] T 00000000000000000000000000000000 P e80f89dd462b42ad8b210a98f0891253 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e80f89dd462b42ad8b210a98f0891253, State: Initialized, Role: FOLLOWER
I20240502 08:02:27.956702 8104 consensus_queue.cc:260] T 00000000000000000000000000000000 P e80f89dd462b42ad8b210a98f0891253 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e80f89dd462b42ad8b210a98f0891253" member_type: VOTER last_known_addr { host: "127.6.204.254" port: 33115 } } peers { permanent_uuid: "1f7d11296f7f4716b9eb5602aa3c842d" member_type: VOTER last_known_addr { host: "127.6.204.253" port: 42483 } } peers { permanent_uuid: "55808e5996304e98b57f3315454e4368" member_type: VOTER last_known_addr { host: "127.6.204.252" port: 46535 } }
I20240502 08:02:27.959048 8251 sys_catalog.cc:455] T 00000000000000000000000000000000 P e80f89dd462b42ad8b210a98f0891253 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e80f89dd462b42ad8b210a98f0891253" member_type: VOTER last_known_addr { host: "127.6.204.254" port: 33115 } } peers { permanent_uuid: "1f7d11296f7f4716b9eb5602aa3c842d" member_type: VOTER last_known_addr { host: "127.6.204.253" port: 42483 } } peers { permanent_uuid: "55808e5996304e98b57f3315454e4368" member_type: VOTER last_known_addr { host: "127.6.204.252" port: 46535 } } }
I20240502 08:02:27.959893 8251 sys_catalog.cc:458] T 00000000000000000000000000000000 P e80f89dd462b42ad8b210a98f0891253 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 08:02:27.960453 8104 sys_catalog.cc:564] T 00000000000000000000000000000000 P e80f89dd462b42ad8b210a98f0891253 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 08:02:27.961334 8235 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55808e5996304e98b57f3315454e4368: Neither blocks nor log segments found. Creating new log.
I20240502 08:02:27.972997 8235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55808e5996304e98b57f3315454e4368: No bootstrap required, opened a new log
I20240502 08:02:27.973968 6963 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
I20240502 08:02:27.974289 6963 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
W20240502 08:02:27.975207 8253 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1f7d11296f7f4716b9eb5602aa3c842d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240502 08:02:27.975528 8253 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240502 08:02:27.975678 8235 raft_consensus.cc:348] T 00000000000000000000000000000000 P 55808e5996304e98b57f3315454e4368 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e80f89dd462b42ad8b210a98f0891253" member_type: VOTER last_known_addr { host: "127.6.204.254" port: 33115 } } peers { permanent_uuid: "1f7d11296f7f4716b9eb5602aa3c842d" member_type: VOTER last_known_addr { host: "127.6.204.253" port: 42483 } } peers { permanent_uuid: "55808e5996304e98b57f3315454e4368" member_type: VOTER last_known_addr { host: "127.6.204.252" port: 46535 } }
I20240502 08:02:27.976291 8235 raft_consensus.cc:374] T 00000000000000000000000000000000 P 55808e5996304e98b57f3315454e4368 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:02:27.976514 8235 raft_consensus.cc:729] T 00000000000000000000000000000000 P 55808e5996304e98b57f3315454e4368 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55808e5996304e98b57f3315454e4368, State: Initialized, Role: FOLLOWER
W20240502 08:02:27.977345 8264 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e80f89dd462b42ad8b210a98f0891253: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20240502 08:02:27.977079 8235 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55808e5996304e98b57f3315454e4368 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e80f89dd462b42ad8b210a98f0891253" member_type: VOTER last_known_addr { host: "127.6.204.254" port: 33115 } } peers { permanent_uuid: "1f7d11296f7f4716b9eb5602aa3c842d" member_type: VOTER last_known_addr { host: "127.6.204.253" port: 42483 } } peers { permanent_uuid: "55808e5996304e98b57f3315454e4368" member_type: VOTER last_known_addr { host: "127.6.204.252" port: 46535 } }
W20240502 08:02:27.977620 8264 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240502 08:02:27.978929 8265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55808e5996304e98b57f3315454e4368 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e80f89dd462b42ad8b210a98f0891253" member_type: VOTER last_known_addr { host: "127.6.204.254" port: 33115 } } peers { permanent_uuid: "1f7d11296f7f4716b9eb5602aa3c842d" member_type: VOTER last_known_addr { host: "127.6.204.253" port: 42483 } } peers { permanent_uuid: "55808e5996304e98b57f3315454e4368" member_type: VOTER last_known_addr { host: "127.6.204.252" port: 46535 } } }
I20240502 08:02:27.979537 8265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55808e5996304e98b57f3315454e4368 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 08:02:27.979974 8235 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55808e5996304e98b57f3315454e4368 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 08:02:27.991261 6963 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:02:27.993918 8276 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 55808e5996304e98b57f3315454e4368: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240502 08:02:27.994328 8276 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20240502 08:02:27.998149 8277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:02:27.999456 8278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:02:28.001179 6963 server_base.cc:1034] running on GCE node
W20240502 08:02:28.001235 8280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:02:28.002311 6963 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:02:28.002526 6963 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:02:28.002712 6963 hybrid_clock.cc:648] HybridClock initialized: now 1714636948002683 us; error 0 us; skew 500 ppm
I20240502 08:02:28.003250 6963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:02:28.005714 6963 webserver.cc:460] Webserver started at http://127.6.204.193:37787/ using document root <none> and password file <none>
I20240502 08:02:28.006234 6963 fs_manager.cc:359] Metadata directory not provided
I20240502 08:02:28.006433 6963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:02:28.006690 6963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:02:28.007896 6963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714636933625983-6963-0/minicluster-data/ts-0-root/instance:
uuid: "b5c5a66357ed43c4973a42e64caa8446"
format_stamp: "Formatted at 2024-05-02 08:02:28 on dist-test-slave-32xs"
I20240502 08:02:28.012760 6963 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240502 08:02:28.015923 8285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:02:28.016723 6963 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 08:02:28.017011 6963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714636933625983-6963-0/minicluster-data/ts-0-root
uuid: "b5c5a66357ed43c4973a42e64caa8446"
format_stamp: "Formatted at 2024-05-02 08:02:28 on dist-test-slave-32xs"
I20240502 08:02:28.017278 6963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714636933625983-6963-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714636933625983-6963-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714636933625983-6963-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:02:28.043404 6963 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:02:28.044663 6963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:02:28.046124 6963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:02:28.048616 6963 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:02:28.048830 6963 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:02:28.049053 6963 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:02:28.049221 6963 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:02:28.067508 8239 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f7d11296f7f4716b9eb5602aa3c842d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:02:28.068109 8239 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f7d11296f7f4716b9eb5602aa3c842d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e80f89dd462b42ad8b210a98f0891253" member_type: VOTER last_known_addr { host: "127.6.204.254" port: 33115 } } peers { permanent_uuid: "1f7d11296f7f4716b9eb5602aa3c842d" member_type: VOTER last_known_addr { host: "127.6.204.253" port: 42483 } } peers { permanent_uuid: "55808e5996304e98b57f3315454e4368" member_type: VOTER last_known_addr { host: "127.6.204.252" port: 46535 } }
I20240502 08:02:28.069757 8239 leader_election.cc:288] T 00000000000000000000000000000000 P 1f7d11296f7f4716b9eb5602aa3c842d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e80f89dd462b42ad8b210a98f0891253 (127.6.204.254:33115), 55808e5996304e98b57f3315454e4368 (127.6.204.252:46535)
I20240502 08:02:28.070636 8079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1f7d11296f7f4716b9eb5602aa3c842d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e80f89dd462b42ad8b210a98f0891253" is_pre_election: true
I20240502 08:02:28.071139 8209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1f7d11296f7f4716b9eb5602aa3c842d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55808e5996304e98b57f3315454e4368" is_pre_election: true
I20240502 08:02:28.071506 8079 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e80f89dd462b42ad8b210a98f0891253 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f7d11296f7f4716b9eb5602aa3c842d in term 0.
I20240502 08:02:28.071789 8209 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 55808e5996304e98b57f3315454e4368 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f7d11296f7f4716b9eb5602aa3c842d in term 0.
I20240502 08:02:28.072825 8121 leader_election.cc:302] T 00000000000000000000000000000000 P 1f7d11296f7f4716b9eb5602aa3c842d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f7d11296f7f4716b9eb5602aa3c842d, e80f89dd462b42ad8b210a98f0891253; no voters:
I20240502 08:02:28.073802 8239 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f7d11296f7f4716b9eb5602aa3c842d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:02:28.074086 8239 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f7d11296f7f4716b9eb5602aa3c842d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:02:28.074400 8239 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f7d11296f7f4716b9eb5602aa3c842d [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:02:28.079156 8239 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f7d11296f7f4716b9eb5602aa3c842d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e80f89dd462b42ad8b210a98f0891253" member_type: VOTER last_known_addr { host: "127.6.204.254" port: 33115 } } peers { permanent_uuid: "1f7d11296f7f4716b9eb5602aa3c842d" member_type: VOTER last_known_addr { host: "127.6.204.253" port: 42483 } } peers { permanent_uuid: "55808e5996304e98b57f3315454e4368" member_type: VOTER last_known_addr { host: "127.6.204.252" port: 46535 } }
I20240502 08:02:28.081182 8239 leader_election.cc:288] T 00000000000000000000000000000000 P 1f7d11296f7f4716b9eb5602aa3c842d [CANDIDATE]: Term 1 election: Requested vote from peers e80f89dd462b42ad8b210a98f0891253 (127.6.204.254:33115), 55808e5996304e98b57f3315454e4368 (127.6.204.252:46535)
I20240502 08:02:28.082299 8209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1f7d11296f7f4716b9eb5602aa3c842d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55808e5996304e98b57f3315454e4368"
I20240502 08:02:28.082453 8079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1f7d11296f7f4716b9eb5602aa3c842d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e80f89dd462b42ad8b210a98f0891253"
I20240502 08:02:28.082914 8209 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55808e5996304e98b57f3315454e4368 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:02:28.083220 8079 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e80f89dd462b42ad8b210a98f0891253 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:02:28.088577 8209 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 55808e5996304e98b57f3315454e4368 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f7d11296f7f4716b9eb5602aa3c842d in term 1.
I20240502 08:02:28.089687 8119 leader_election.cc:302] T 00000000000000000000000000000000 P 1f7d11296f7f4716b9eb5602aa3c842d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f7d11296f7f4716b9eb5602aa3c842d, 55808e5996304e98b57f3315454e4368; no voters:
I20240502 08:02:28.090592 8239 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f7d11296f7f4716b9eb5602aa3c842d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:02:28.091194 8079 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e80f89dd462b42ad8b210a98f0891253 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f7d11296f7f4716b9eb5602aa3c842d in term 1.
I20240502 08:02:28.093216 8239 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1f7d11296f7f4716b9eb5602aa3c842d [term 1 LEADER]: Becoming Leader. State: Replica: 1f7d11296f7f4716b9eb5602aa3c842d, State: Running, Role: LEADER
I20240502 08:02:28.093864 8239 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f7d11296f7f4716b9eb5602aa3c842d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e80f89dd462b42ad8b210a98f0891253" member_type: VOTER last_known_addr { host: "127.6.204.254" port: 33115 } } peers { permanent_uuid: "1f7d11296f7f4716b9eb5602aa3c842d" member_type: VOTER last_known_addr { host: "127.6.204.253" port: 42483 } } peers { permanent_uuid: "55808e5996304e98b57f3315454e4368" member_type: VOTER last_known_addr { host: "127.6.204.252" port: 46535 } }
I20240502 08:02:28.099128 8337 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f7d11296f7f4716b9eb5602aa3c842d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f7d11296f7f4716b9eb5602aa3c842d. Latest consensus state: current_term: 1 leader_uuid: "1f7d11296f7f4716b9eb5602aa3c842d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e80f89dd462b42ad8b210a98f0891253" member_type: VOTER last_known_addr { host: "127.6.204.254" port: 33115 } } peers { permanent_uuid: "1f7d11296f7f4716b9eb5602aa3c842d" member_type: VOTER last_known_addr { host: "127.6.204.253" port: 42483 } } peers { permanent_uuid: "55808e5996304e98b57f3315454e4368" member_type: VOTER last_known_addr { host: "127.6.204.252" port: 46535 } } }
I20240502 08:02:28.100157 8337 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f7d11296f7f4716b9eb5602aa3c842d [sys.catalog]: This master's current role is: LEADER
I20240502 08:02:28.101840 8346 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 08:02:28.104691 6963 rpc_server.cc:307] RPC server started. Bound to: 127.6.204.193:35847
I20240502 08:02:28.104869 8350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.204.193:35847 every 8 connection(s)
I20240502 08:02:28.107846 8346 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 08:02:28.127923 8209 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 55808e5996304e98b57f3315454e4368 [term 1 FOLLOWER]: Refusing update from remote peer 1f7d11296f7f4716b9eb5602aa3c842d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:02:28.129850 8079 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e80f89dd462b42ad8b210a98f0891253 [term 1 FOLLOWER]: Refusing update from remote peer 1f7d11296f7f4716b9eb5602aa3c842d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:02:28.132349 8351 heartbeater.cc:347] Connected to a master server at 127.6.204.253:42483
I20240502 08:02:28.132778 8351 heartbeater.cc:464] Registering TS with master...
I20240502 08:02:28.132534 8337 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1f7d11296f7f4716b9eb5602aa3c842d [LEADER]: Connected to new peer: Peer: permanent_uuid: "55808e5996304e98b57f3315454e4368" member_type: VOTER last_known_addr { host: "127.6.204.252" port: 46535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:02:28.133652 8351 heartbeater.cc:511] Master 127.6.204.253:42483 requested a full tablet report, sending...
I20240502 08:02:28.135412 8337 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1f7d11296f7f4716b9eb5602aa3c842d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e80f89dd462b42ad8b210a98f0891253" member_type: VOTER last_known_addr { host: "127.6.204.254" port: 33115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:02:28.136103 8134 ts_manager.cc:194] Registered new tserver with Master: b5c5a66357ed43c4973a42e64caa8446 (127.6.204.193:35847)
I20240502 08:02:28.138767 6963 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 08:02:28.144917 8352 heartbeater.cc:347] Connected to a master server at 127.6.204.252:46535
I20240502 08:02:28.145450 8352 heartbeater.cc:464] Registering TS with master...
I20240502 08:02:28.146854 8352 heartbeater.cc:511] Master 127.6.204.252:46535 requested a full tablet report, sending...
I20240502 08:02:28.153201 8353 heartbeater.cc:347] Connected to a master server at 127.6.204.254:33115
I20240502 08:02:28.153717 8353 heartbeater.cc:464] Registering TS with master...
I20240502 08:02:28.155066 8353 heartbeater.cc:511] Master 127.6.204.254:33115 requested a full tablet report, sending...
W20240502 08:02:28.161379 8360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:02:28.162506 8199 ts_manager.cc:194] Registered new tserver with Master: b5c5a66357ed43c4973a42e64caa8446 (127.6.204.193:35847)
I20240502 08:02:28.162947 8265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55808e5996304e98b57f3315454e4368 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f7d11296f7f4716b9eb5602aa3c842d. Latest consensus state: current_term: 1 leader_uuid: "1f7d11296f7f4716b9eb5602aa3c842d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e80f89dd462b42ad8b210a98f0891253" member_type: VOTER last_known_addr { host: "127.6.204.254" port: 33115 } } peers { permanent_uuid: "1f7d11296f7f4716b9eb5602aa3c842d" member_type: VOTER last_known_addr { host: "127.6.204.253" port: 42483 } } peers { permanent_uuid: "55808e5996304e98b57f3315454e4368" member_type: VOTER last_known_addr { host: "127.6.204.252" port: 46535 } } }
I20240502 08:02:28.163687 8265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55808e5996304e98b57f3315454e4368 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 08:02:28.168114 8069 ts_manager.cc:194] Registered new tserver with Master: b5c5a66357ed43c4973a42e64caa8446 (127.6.204.193:35847)
I20240502 08:02:28.171244 8251 sys_catalog.cc:455] T 00000000000000000000000000000000 P e80f89dd462b42ad8b210a98f0891253 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f7d11296f7f4716b9eb5602aa3c842d. Latest consensus state: current_term: 1 leader_uuid: "1f7d11296f7f4716b9eb5602aa3c842d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e80f89dd462b42ad8b210a98f0891253" member_type: VOTER last_known_addr { host: "127.6.204.254" port: 33115 } } peers { permanent_uuid: "1f7d11296f7f4716b9eb5602aa3c842d" member_type: VOTER last_known_addr { host: "127.6.204.253" port: 42483 } } peers { permanent_uuid: "55808e5996304e98b57f3315454e4368" member_type: VOTER last_known_addr { host: "127.6.204.252" port: 46535 } } }
I20240502 08:02:28.172281 8251 sys_catalog.cc:458] T 00000000000000000000000000000000 P e80f89dd462b42ad8b210a98f0891253 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 08:02:28.177700 8346 catalog_manager.cc:1347] Generated new cluster ID: 9691e79e85f64e07b05f9266f9654bb6
I20240502 08:02:28.178074 8346 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
W20240502 08:02:28.178572 8364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:02:28.199628 8265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55808e5996304e98b57f3315454e4368 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1f7d11296f7f4716b9eb5602aa3c842d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e80f89dd462b42ad8b210a98f0891253" member_type: VOTER last_known_addr { host: "127.6.204.254" port: 33115 } } peers { permanent_uuid: "1f7d11296f7f4716b9eb5602aa3c842d" member_type: VOTER last_known_addr { host: "127.6.204.253" port: 42483 } } peers { permanent_uuid: "55808e5996304e98b57f3315454e4368" member_type: VOTER last_known_addr { host: "127.6.204.252" port: 46535 } } }
I20240502 08:02:28.201092 8265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55808e5996304e98b57f3315454e4368 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 08:02:28.204262 8239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f7d11296f7f4716b9eb5602aa3c842d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1f7d11296f7f4716b9eb5602aa3c842d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e80f89dd462b42ad8b210a98f0891253" member_type: VOTER last_known_addr { host: "127.6.204.254" port: 33115 } } peers { permanent_uuid: "1f7d11296f7f4716b9eb5602aa3c842d" member_type: VOTER last_known_addr { host: "127.6.204.253" port: 42483 } } peers { permanent_uuid: "55808e5996304e98b57f3315454e4368" member_type: VOTER last_known_addr { host: "127.6.204.252" port: 46535 } } }
I20240502 08:02:28.205068 8239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f7d11296f7f4716b9eb5602aa3c842d [sys.catalog]: This master's current role is: LEADER
I20240502 08:02:28.204015 8337 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f7d11296f7f4716b9eb5602aa3c842d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1f7d11296f7f4716b9eb5602aa3c842d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e80f89dd462b42ad8b210a98f0891253" member_type: VOTER last_known_addr { host: "127.6.204.254" port: 33115 } } peers { permanent_uuid: "1f7d11296f7f4716b9eb5602aa3c842d" member_type: VOTER last_known_addr { host: "127.6.204.253" port: 42483 } } peers { permanent_uuid: "55808e5996304e98b57f3315454e4368" member_type: VOTER last_known_addr { host: "127.6.204.252" port: 46535 } } }
I20240502 08:02:28.206488 8251 sys_catalog.cc:455] T 00000000000000000000000000000000 P e80f89dd462b42ad8b210a98f0891253 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1f7d11296f7f4716b9eb5602aa3c842d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e80f89dd462b42ad8b210a98f0891253" member_type: VOTER last_known_addr { host: "127.6.204.254" port: 33115 } } peers { permanent_uuid: "1f7d11296f7f4716b9eb5602aa3c842d" member_type: VOTER last_known_addr { host: "127.6.204.253" port: 42483 } } peers { permanent_uuid: "55808e5996304e98b57f3315454e4368" member_type: VOTER last_known_addr { host: "127.6.204.252" port: 46535 } } }
I20240502 08:02:28.207497 8251 sys_catalog.cc:458] T 00000000000000000000000000000000 P e80f89dd462b42ad8b210a98f0891253 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 08:02:28.205626 8337 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f7d11296f7f4716b9eb5602aa3c842d [sys.catalog]: This master's current role is: LEADER
I20240502 08:02:28.217519 8346 catalog_manager.cc:1370] Generated new certificate authority record
W20240502 08:02:28.217474 8370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:02:28.220183 8346 catalog_manager.cc:1504] Loading token signing keys...
I20240502 08:02:28.220675 6963 server_base.cc:1034] running on GCE node
I20240502 08:02:28.221748 6963 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:02:28.222020 6963 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:02:28.222263 6963 hybrid_clock.cc:648] HybridClock initialized: now 1714636948222242 us; error 0 us; skew 500 ppm
I20240502 08:02:28.222952 6963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:02:28.225944 6963 webserver.cc:460] Webserver started at http://127.6.204.194:41447/ using document root <none> and password file <none>
I20240502 08:02:28.226539 6963 fs_manager.cc:359] Metadata directory not provided
I20240502 08:02:28.226768 6963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:02:28.227051 6963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:02:28.228317 6963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714636933625983-6963-0/minicluster-data/ts-1-root/instance:
uuid: "d35dbf5f483f45199c94cf946704a35a"
format_stamp: "Formatted at 2024-05-02 08:02:28 on dist-test-slave-32xs"
I20240502 08:02:28.233948 6963 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240502 08:02:28.238488 8375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:02:28.239385 6963 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20240502 08:02:28.239686 6963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714636933625983-6963-0/minicluster-data/ts-1-root
uuid: "d35dbf5f483f45199c94cf946704a35a"
format_stamp: "Formatted at 2024-05-02 08:02:28 on dist-test-slave-32xs"
I20240502 08:02:28.240027 6963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714636933625983-6963-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714636933625983-6963-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714636933625983-6963-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:02:28.245826 8346 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 1f7d11296f7f4716b9eb5602aa3c842d: Generated new TSK 0
I20240502 08:02:28.246531 8346 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 08:02:28.259687 6963 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:02:28.261169 6963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:02:28.262526 6963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:02:28.264766 6963 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:02:28.264948 6963 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:02:28.265201 6963 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:02:28.265347 6963 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:02:28.305912 6963 rpc_server.cc:307] RPC server started. Bound to: 127.6.204.194:41967
I20240502 08:02:28.306012 8437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.204.194:41967 every 8 connection(s)
I20240502 08:02:28.315291 6963 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 08:02:28.326092 8438 heartbeater.cc:347] Connected to a master server at 127.6.204.253:42483
I20240502 08:02:28.326581 8438 heartbeater.cc:464] Registering TS with master...
I20240502 08:02:28.327486 8438 heartbeater.cc:511] Master 127.6.204.253:42483 requested a full tablet report, sending...
I20240502 08:02:28.329753 8134 ts_manager.cc:194] Registered new tserver with Master: d35dbf5f483f45199c94cf946704a35a (127.6.204.194:41967)
I20240502 08:02:28.332087 8134 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49578
I20240502 08:02:28.340997 8440 heartbeater.cc:347] Connected to a master server at 127.6.204.254:33115
I20240502 08:02:28.341514 8440 heartbeater.cc:464] Registering TS with master...
I20240502 08:02:28.342414 8440 heartbeater.cc:511] Master 127.6.204.254:33115 requested a full tablet report, sending...
I20240502 08:02:28.346027 8069 ts_manager.cc:194] Registered new tserver with Master: d35dbf5f483f45199c94cf946704a35a (127.6.204.194:41967)
I20240502 08:02:28.342692 8439 heartbeater.cc:347] Connected to a master server at 127.6.204.252:46535
I20240502 08:02:28.348351 8439 heartbeater.cc:464] Registering TS with master...
I20240502 08:02:28.349292 8439 heartbeater.cc:511] Master 127.6.204.252:46535 requested a full tablet report, sending...
W20240502 08:02:28.351902 8446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:02:28.353049 8445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:02:28.355861 6963 server_base.cc:1034] running on GCE node
W20240502 08:02:28.356818 8448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:02:28.357831 6963 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:02:28.358103 6963 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:02:28.358326 6963 hybrid_clock.cc:648] HybridClock initialized: now 1714636948358309 us; error 0 us; skew 500 ppm
I20240502 08:02:28.358996 6963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:02:28.362946 8199 ts_manager.cc:194] Registered new tserver with Master: d35dbf5f483f45199c94cf946704a35a (127.6.204.194:41967)
I20240502 08:02:28.364080 6963 webserver.cc:460] Webserver started at http://127.6.204.195:34081/ using document root <none> and password file <none>
I20240502 08:02:28.364737 6963 fs_manager.cc:359] Metadata directory not provided
I20240502 08:02:28.364996 6963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:02:28.365335 6963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:02:28.366801 6963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714636933625983-6963-0/minicluster-data/ts-2-root/instance:
uuid: "ecf2a63dfa304fe48ecc58b49ce204ff"
format_stamp: "Formatted at 2024-05-02 08:02:28 on dist-test-slave-32xs"
I20240502 08:02:28.371140 6963 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240502 08:02:28.374372 8453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:02:28.375232 6963 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 08:02:28.375528 6963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714636933625983-6963-0/minicluster-data/ts-2-root
uuid: "ecf2a63dfa304fe48ecc58b49ce204ff"
format_stamp: "Formatted at 2024-05-02 08:02:28 on dist-test-slave-32xs"
I20240502 08:02:28.375835 6963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714636933625983-6963-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714636933625983-6963-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714636933625983-6963-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:02:28.387763 6963 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:02:28.388991 6963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:02:28.390451 6963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:02:28.392866 6963 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:02:28.393105 6963 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:02:28.393337 6963 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:02:28.393496 6963 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:02:28.433429 6963 rpc_server.cc:307] RPC server started. Bound to: 127.6.204.195:41329
I20240502 08:02:28.433506 8515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.204.195:41329 every 8 connection(s)
I20240502 08:02:28.452263 8516 heartbeater.cc:347] Connected to a master server at 127.6.204.253:42483
I20240502 08:02:28.452733 8516 heartbeater.cc:464] Registering TS with master...
I20240502 08:02:28.453094 8518 heartbeater.cc:347] Connected to a master server at 127.6.204.254:33115
I20240502 08:02:28.453444 8518 heartbeater.cc:464] Registering TS with master...
I20240502 08:02:28.453586 8516 heartbeater.cc:511] Master 127.6.204.253:42483 requested a full tablet report, sending...
I20240502 08:02:28.454118 8517 heartbeater.cc:347] Connected to a master server at 127.6.204.252:46535
I20240502 08:02:28.454286 8518 heartbeater.cc:511] Master 127.6.204.254:33115 requested a full tablet report, sending...
I20240502 08:02:28.454525 8517 heartbeater.cc:464] Registering TS with master...
I20240502 08:02:28.455159 8517 heartbeater.cc:511] Master 127.6.204.252:46535 requested a full tablet report, sending...
I20240502 08:02:28.455997 8134 ts_manager.cc:194] Registered new tserver with Master: ecf2a63dfa304fe48ecc58b49ce204ff (127.6.204.195:41329)
I20240502 08:02:28.456961 8069 ts_manager.cc:194] Registered new tserver with Master: ecf2a63dfa304fe48ecc58b49ce204ff (127.6.204.195:41329)
I20240502 08:02:28.457672 8134 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49582
I20240502 08:02:28.457731 8199 ts_manager.cc:194] Registered new tserver with Master: ecf2a63dfa304fe48ecc58b49ce204ff (127.6.204.195:41329)
I20240502 08:02:28.458714 6963 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.017242549s
I20240502 08:02:28.981359 8264 catalog_manager.cc:1259] Loaded cluster ID: 9691e79e85f64e07b05f9266f9654bb6
I20240502 08:02:28.981637 8264 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e80f89dd462b42ad8b210a98f0891253: loading cluster ID for follower catalog manager: success
I20240502 08:02:28.985978 8264 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e80f89dd462b42ad8b210a98f0891253: acquiring CA information for follower catalog manager: success
I20240502 08:02:28.989037 8264 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e80f89dd462b42ad8b210a98f0891253: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240502 08:02:28.997881 8276 catalog_manager.cc:1259] Loaded cluster ID: 9691e79e85f64e07b05f9266f9654bb6
I20240502 08:02:28.998199 8276 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 55808e5996304e98b57f3315454e4368: loading cluster ID for follower catalog manager: success
I20240502 08:02:29.002712 8276 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 55808e5996304e98b57f3315454e4368: acquiring CA information for follower catalog manager: success
I20240502 08:02:29.005908 8276 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 55808e5996304e98b57f3315454e4368: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240502 08:02:29.150794 8134 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49574
I20240502 08:02:29.335392 8438 heartbeater.cc:503] Master 127.6.204.253:42483 was elected leader, sending a full tablet report...
I20240502 08:02:29.460424 8516 heartbeater.cc:503] Master 127.6.204.253:42483 was elected leader, sending a full tablet report...
I20240502 08:02:29.963382 8250 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:02:30.018571 6963 test_util.cc:268] Using random seed: 106109429
I20240502 08:02:30.058537 8134 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34430:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\252\252\252*\004\001\000TUUU""\004\001\000\252\252\252*\004\001\000TUUU"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:02:30.144843 8481 tablet_service.cc:1467] Processing CreateTablet for tablet 48160b0a72244bbda66c5a5f927379d3 (DEFAULT_TABLE table=test-workload [id=60408b8ba1c74955a5c3bade6d8e423d]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20240502 08:02:30.146384 8481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48160b0a72244bbda66c5a5f927379d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:02:30.146934 8313 tablet_service.cc:1467] Processing CreateTablet for tablet 42c8e173d0da466da778ec873c9dc745 (DEFAULT_TABLE table=test-workload [id=60408b8ba1c74955a5c3bade6d8e423d]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20240502 08:02:30.147241 8403 tablet_service.cc:1467] Processing CreateTablet for tablet 5f82424c85be4acab0d3a5bc3ed841c8 (DEFAULT_TABLE table=test-workload [id=60408b8ba1c74955a5c3bade6d8e423d]), partition=RANGE (key) PARTITION VALUES < 715827882
I20240502 08:02:30.147998 8313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42c8e173d0da466da778ec873c9dc745. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:02:30.148589 8403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f82424c85be4acab0d3a5bc3ed841c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:02:30.153218 8351 heartbeater.cc:503] Master 127.6.204.253:42483 was elected leader, sending a full tablet report...
I20240502 08:02:30.163225 8552 tablet_bootstrap.cc:492] T 42c8e173d0da466da778ec873c9dc745 P b5c5a66357ed43c4973a42e64caa8446: Bootstrap starting.
I20240502 08:02:30.166883 8554 tablet_bootstrap.cc:492] T 5f82424c85be4acab0d3a5bc3ed841c8 P d35dbf5f483f45199c94cf946704a35a: Bootstrap starting.
I20240502 08:02:30.168884 8552 tablet_bootstrap.cc:654] T 42c8e173d0da466da778ec873c9dc745 P b5c5a66357ed43c4973a42e64caa8446: Neither blocks nor log segments found. Creating new log.
I20240502 08:02:30.170387 8553 tablet_bootstrap.cc:492] T 48160b0a72244bbda66c5a5f927379d3 P ecf2a63dfa304fe48ecc58b49ce204ff: Bootstrap starting.
I20240502 08:02:30.174036 8554 tablet_bootstrap.cc:654] T 5f82424c85be4acab0d3a5bc3ed841c8 P d35dbf5f483f45199c94cf946704a35a: Neither blocks nor log segments found. Creating new log.
I20240502 08:02:30.175155 8552 tablet_bootstrap.cc:492] T 42c8e173d0da466da778ec873c9dc745 P b5c5a66357ed43c4973a42e64caa8446: No bootstrap required, opened a new log
I20240502 08:02:30.175835 8552 ts_tablet_manager.cc:1367] T 42c8e173d0da466da778ec873c9dc745 P b5c5a66357ed43c4973a42e64caa8446: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.009s
I20240502 08:02:30.178084 8553 tablet_bootstrap.cc:654] T 48160b0a72244bbda66c5a5f927379d3 P ecf2a63dfa304fe48ecc58b49ce204ff: Neither blocks nor log segments found. Creating new log.
I20240502 08:02:30.179628 8552 raft_consensus.cc:348] T 42c8e173d0da466da778ec873c9dc745 P b5c5a66357ed43c4973a42e64caa8446 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5c5a66357ed43c4973a42e64caa8446" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 35847 } }
I20240502 08:02:30.180284 8552 raft_consensus.cc:374] T 42c8e173d0da466da778ec873c9dc745 P b5c5a66357ed43c4973a42e64caa8446 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:02:30.180532 8552 raft_consensus.cc:729] T 42c8e173d0da466da778ec873c9dc745 P b5c5a66357ed43c4973a42e64caa8446 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5c5a66357ed43c4973a42e64caa8446, State: Initialized, Role: FOLLOWER
I20240502 08:02:30.181121 8552 consensus_queue.cc:260] T 42c8e173d0da466da778ec873c9dc745 P b5c5a66357ed43c4973a42e64caa8446 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5c5a66357ed43c4973a42e64caa8446" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 35847 } }
I20240502 08:02:30.181828 8554 tablet_bootstrap.cc:492] T 5f82424c85be4acab0d3a5bc3ed841c8 P d35dbf5f483f45199c94cf946704a35a: No bootstrap required, opened a new log
I20240502 08:02:30.181888 8552 raft_consensus.cc:388] T 42c8e173d0da466da778ec873c9dc745 P b5c5a66357ed43c4973a42e64caa8446 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:02:30.182376 8554 ts_tablet_manager.cc:1367] T 5f82424c85be4acab0d3a5bc3ed841c8 P d35dbf5f483f45199c94cf946704a35a: Time spent bootstrapping tablet: real 0.016s user 0.013s sys 0.000s
I20240502 08:02:30.182399 8552 raft_consensus.cc:482] T 42c8e173d0da466da778ec873c9dc745 P b5c5a66357ed43c4973a42e64caa8446 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:02:30.182945 8552 raft_consensus.cc:3037] T 42c8e173d0da466da778ec873c9dc745 P b5c5a66357ed43c4973a42e64caa8446 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:02:30.183938 8553 tablet_bootstrap.cc:492] T 48160b0a72244bbda66c5a5f927379d3 P ecf2a63dfa304fe48ecc58b49ce204ff: No bootstrap required, opened a new log
I20240502 08:02:30.184587 8553 ts_tablet_manager.cc:1367] T 48160b0a72244bbda66c5a5f927379d3 P ecf2a63dfa304fe48ecc58b49ce204ff: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
I20240502 08:02:30.185477 8554 raft_consensus.cc:348] T 5f82424c85be4acab0d3a5bc3ed841c8 P d35dbf5f483f45199c94cf946704a35a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d35dbf5f483f45199c94cf946704a35a" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 41967 } }
I20240502 08:02:30.186095 8554 raft_consensus.cc:374] T 5f82424c85be4acab0d3a5bc3ed841c8 P d35dbf5f483f45199c94cf946704a35a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:02:30.186424 8554 raft_consensus.cc:729] T 5f82424c85be4acab0d3a5bc3ed841c8 P d35dbf5f483f45199c94cf946704a35a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35dbf5f483f45199c94cf946704a35a, State: Initialized, Role: FOLLOWER
I20240502 08:02:30.187114 8554 consensus_queue.cc:260] T 5f82424c85be4acab0d3a5bc3ed841c8 P d35dbf5f483f45199c94cf946704a35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d35dbf5f483f45199c94cf946704a35a" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 41967 } }
I20240502 08:02:30.187502 8552 raft_consensus.cc:504] T 42c8e173d0da466da778ec873c9dc745 P b5c5a66357ed43c4973a42e64caa8446 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5c5a66357ed43c4973a42e64caa8446" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 35847 } }
I20240502 08:02:30.187407 8553 raft_consensus.cc:348] T 48160b0a72244bbda66c5a5f927379d3 P ecf2a63dfa304fe48ecc58b49ce204ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecf2a63dfa304fe48ecc58b49ce204ff" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 41329 } }
I20240502 08:02:30.187800 8554 raft_consensus.cc:388] T 5f82424c85be4acab0d3a5bc3ed841c8 P d35dbf5f483f45199c94cf946704a35a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:02:30.188154 8553 raft_consensus.cc:374] T 48160b0a72244bbda66c5a5f927379d3 P ecf2a63dfa304fe48ecc58b49ce204ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:02:30.188270 8554 raft_consensus.cc:482] T 5f82424c85be4acab0d3a5bc3ed841c8 P d35dbf5f483f45199c94cf946704a35a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:02:30.188236 8552 leader_election.cc:302] T 42c8e173d0da466da778ec873c9dc745 P b5c5a66357ed43c4973a42e64caa8446 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5c5a66357ed43c4973a42e64caa8446; no voters:
I20240502 08:02:30.188634 8553 raft_consensus.cc:729] T 48160b0a72244bbda66c5a5f927379d3 P ecf2a63dfa304fe48ecc58b49ce204ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecf2a63dfa304fe48ecc58b49ce204ff, State: Initialized, Role: FOLLOWER
I20240502 08:02:30.188827 8554 raft_consensus.cc:3037] T 5f82424c85be4acab0d3a5bc3ed841c8 P d35dbf5f483f45199c94cf946704a35a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:02:30.189819 8552 leader_election.cc:288] T 42c8e173d0da466da778ec873c9dc745 P b5c5a66357ed43c4973a42e64caa8446 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:02:30.190363 8558 raft_consensus.cc:2781] T 42c8e173d0da466da778ec873c9dc745 P b5c5a66357ed43c4973a42e64caa8446 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:02:30.190228 8553 consensus_queue.cc:260] T 48160b0a72244bbda66c5a5f927379d3 P ecf2a63dfa304fe48ecc58b49ce204ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecf2a63dfa304fe48ecc58b49ce204ff" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 41329 } }
I20240502 08:02:30.191056 8553 raft_consensus.cc:388] T 48160b0a72244bbda66c5a5f927379d3 P ecf2a63dfa304fe48ecc58b49ce204ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:02:30.191471 8553 raft_consensus.cc:482] T 48160b0a72244bbda66c5a5f927379d3 P ecf2a63dfa304fe48ecc58b49ce204ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:02:30.191915 8553 raft_consensus.cc:3037] T 48160b0a72244bbda66c5a5f927379d3 P ecf2a63dfa304fe48ecc58b49ce204ff [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:02:30.195376 8558 raft_consensus.cc:686] T 42c8e173d0da466da778ec873c9dc745 P b5c5a66357ed43c4973a42e64caa8446 [term 1 LEADER]: Becoming Leader. State: Replica: b5c5a66357ed43c4973a42e64caa8446, State: Running, Role: LEADER
I20240502 08:02:30.196156 8558 consensus_queue.cc:237] T 42c8e173d0da466da778ec873c9dc745 P b5c5a66357ed43c4973a42e64caa8446 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5c5a66357ed43c4973a42e64caa8446" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 35847 } }
I20240502 08:02:30.197536 8554 raft_consensus.cc:504] T 5f82424c85be4acab0d3a5bc3ed841c8 P d35dbf5f483f45199c94cf946704a35a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d35dbf5f483f45199c94cf946704a35a" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 41967 } }
I20240502 08:02:30.198377 8554 leader_election.cc:302] T 5f82424c85be4acab0d3a5bc3ed841c8 P d35dbf5f483f45199c94cf946704a35a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d35dbf5f483f45199c94cf946704a35a; no voters:
I20240502 08:02:30.201509 8552 ts_tablet_manager.cc:1398] T 42c8e173d0da466da778ec873c9dc745 P b5c5a66357ed43c4973a42e64caa8446: Time spent starting tablet: real 0.025s user 0.009s sys 0.012s
I20240502 08:02:30.203393 8553 raft_consensus.cc:504] T 48160b0a72244bbda66c5a5f927379d3 P ecf2a63dfa304fe48ecc58b49ce204ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecf2a63dfa304fe48ecc58b49ce204ff" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 41329 } }
I20240502 08:02:30.204190 8553 leader_election.cc:302] T 48160b0a72244bbda66c5a5f927379d3 P ecf2a63dfa304fe48ecc58b49ce204ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ecf2a63dfa304fe48ecc58b49ce204ff; no voters:
I20240502 08:02:30.209908 8134 catalog_manager.cc:5484] T 42c8e173d0da466da778ec873c9dc745 P b5c5a66357ed43c4973a42e64caa8446 reported cstate change: term changed from 0 to 1, leader changed from <none> to b5c5a66357ed43c4973a42e64caa8446 (127.6.204.193). New cstate: current_term: 1 leader_uuid: "b5c5a66357ed43c4973a42e64caa8446" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5c5a66357ed43c4973a42e64caa8446" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 35847 } health_report { overall_health: HEALTHY } } }
I20240502 08:02:30.222064 8553 leader_election.cc:288] T 48160b0a72244bbda66c5a5f927379d3 P ecf2a63dfa304fe48ecc58b49ce204ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:02:30.222687 8563 raft_consensus.cc:2781] T 48160b0a72244bbda66c5a5f927379d3 P ecf2a63dfa304fe48ecc58b49ce204ff [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:02:30.227216 8563 raft_consensus.cc:686] T 48160b0a72244bbda66c5a5f927379d3 P ecf2a63dfa304fe48ecc58b49ce204ff [term 1 LEADER]: Becoming Leader. State: Replica: ecf2a63dfa304fe48ecc58b49ce204ff, State: Running, Role: LEADER
I20240502 08:02:30.228302 8563 consensus_queue.cc:237] T 48160b0a72244bbda66c5a5f927379d3 P ecf2a63dfa304fe48ecc58b49ce204ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecf2a63dfa304fe48ecc58b49ce204ff" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 41329 } }
I20240502 08:02:30.235041 8553 ts_tablet_manager.cc:1398] T 48160b0a72244bbda66c5a5f927379d3 P ecf2a63dfa304fe48ecc58b49ce204ff: Time spent starting tablet: real 0.050s user 0.027s sys 0.013s
I20240502 08:02:30.250972 8554 leader_election.cc:288] T 5f82424c85be4acab0d3a5bc3ed841c8 P d35dbf5f483f45199c94cf946704a35a [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:02:30.251406 8562 raft_consensus.cc:2781] T 5f82424c85be4acab0d3a5bc3ed841c8 P d35dbf5f483f45199c94cf946704a35a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:02:30.253870 8554 ts_tablet_manager.cc:1398] T 5f82424c85be4acab0d3a5bc3ed841c8 P d35dbf5f483f45199c94cf946704a35a: Time spent starting tablet: real 0.071s user 0.034s sys 0.024s
I20240502 08:02:30.253662 8134 catalog_manager.cc:5484] T 48160b0a72244bbda66c5a5f927379d3 P ecf2a63dfa304fe48ecc58b49ce204ff reported cstate change: term changed from 0 to 1, leader changed from <none> to ecf2a63dfa304fe48ecc58b49ce204ff (127.6.204.195). New cstate: current_term: 1 leader_uuid: "ecf2a63dfa304fe48ecc58b49ce204ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecf2a63dfa304fe48ecc58b49ce204ff" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 41329 } health_report { overall_health: HEALTHY } } }
I20240502 08:02:30.257745 8562 raft_consensus.cc:686] T 5f82424c85be4acab0d3a5bc3ed841c8 P d35dbf5f483f45199c94cf946704a35a [term 1 LEADER]: Becoming Leader. State: Replica: d35dbf5f483f45199c94cf946704a35a, State: Running, Role: LEADER
I20240502 08:02:30.258481 8562 consensus_queue.cc:237] T 5f82424c85be4acab0d3a5bc3ed841c8 P d35dbf5f483f45199c94cf946704a35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d35dbf5f483f45199c94cf946704a35a" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 41967 } }
I20240502 08:02:30.290378 8132 catalog_manager.cc:5484] T 5f82424c85be4acab0d3a5bc3ed841c8 P d35dbf5f483f45199c94cf946704a35a reported cstate change: term changed from 0 to 1, leader changed from <none> to d35dbf5f483f45199c94cf946704a35a (127.6.204.194). New cstate: current_term: 1 leader_uuid: "d35dbf5f483f45199c94cf946704a35a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d35dbf5f483f45199c94cf946704a35a" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 41967 } health_report { overall_health: HEALTHY } } }
I20240502 08:02:30.332501 6963 master.cc:537] Master@127.6.204.253:42483 shutting down...
I20240502 08:02:30.354648 6963 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 1f7d11296f7f4716b9eb5602aa3c842d [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:02:30.355471 6963 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 1f7d11296f7f4716b9eb5602aa3c842d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:02:30.355819 6963 tablet_replica.cc:330] T 00000000000000000000000000000000 P 1f7d11296f7f4716b9eb5602aa3c842d: stopping tablet replica
I20240502 08:02:30.377296 6963 master.cc:552] Master@127.6.204.253:42483 shutdown complete.
W20240502 08:02:31.306936 8351 heartbeater.cc:644] Failed to heartbeat to 127.6.204.253:42483 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.6.204.253:42483: connect: Connection refused (error 111)
I20240502 08:02:31.814846 8577 raft_consensus.cc:482] T 00000000000000000000000000000000 P e80f89dd462b42ad8b210a98f0891253 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1f7d11296f7f4716b9eb5602aa3c842d)
I20240502 08:02:31.815366 8577 raft_consensus.cc:504] T 00000000000000000000000000000000 P e80f89dd462b42ad8b210a98f0891253 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e80f89dd462b42ad8b210a98f0891253" member_type: VOTER last_known_addr { host: "127.6.204.254" port: 33115 } } peers { permanent_uuid: "1f7d11296f7f4716b9eb5602aa3c842d" member_type: VOTER last_known_addr { host: "127.6.204.253" port: 42483 } } peers { permanent_uuid: "55808e5996304e98b57f3315454e4368" member_type: VOTER last_known_addr { host: "127.6.204.252" port: 46535 } }
I20240502 08:02:31.817267 8577 leader_election.cc:288] T 00000000000000000000000000000000 P e80f89dd462b42ad8b210a98f0891253 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1f7d11296f7f4716b9eb5602aa3c842d (127.6.204.253:42483), 55808e5996304e98b57f3315454e4368 (127.6.204.252:46535)
I20240502 08:02:31.817420 8578 raft_consensus.cc:482] T 00000000000000000000000000000000 P 55808e5996304e98b57f3315454e4368 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1f7d11296f7f4716b9eb5602aa3c842d)
I20240502 08:02:31.818156 8578 raft_consensus.cc:504] T 00000000000000000000000000000000 P 55808e5996304e98b57f3315454e4368 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e80f89dd462b42ad8b210a98f0891253" member_type: VOTER last_known_addr { host: "127.6.204.254" port: 33115 } } peers { permanent_uuid: "1f7d11296f7f4716b9eb5602aa3c842d" member_type: VOTER last_known_addr { host: "127.6.204.253" port: 42483 } } peers { permanent_uuid: "55808e5996304e98b57f3315454e4368" member_type: VOTER last_known_addr { host: "127.6.204.252" port: 46535 } }
I20240502 08:02:31.818559 8208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e80f89dd462b42ad8b210a98f0891253" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "55808e5996304e98b57f3315454e4368" is_pre_election: true
I20240502 08:02:31.819279 8208 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 55808e5996304e98b57f3315454e4368 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e80f89dd462b42ad8b210a98f0891253 in term 1.
I20240502 08:02:31.821255 8054 leader_election.cc:302] T 00000000000000000000000000000000 P e80f89dd462b42ad8b210a98f0891253 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55808e5996304e98b57f3315454e4368, e80f89dd462b42ad8b210a98f0891253; no voters:
I20240502 08:02:31.821105 8078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "55808e5996304e98b57f3315454e4368" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "e80f89dd462b42ad8b210a98f0891253" is_pre_election: true
I20240502 08:02:31.822033 8577 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e80f89dd462b42ad8b210a98f0891253 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:02:31.822568 8078 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e80f89dd462b42ad8b210a98f0891253 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55808e5996304e98b57f3315454e4368 in term 1.
I20240502 08:02:31.823422 8578 leader_election.cc:288] T 00000000000000000000000000000000 P 55808e5996304e98b57f3315454e4368 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e80f89dd462b42ad8b210a98f0891253 (127.6.204.254:33115), 1f7d11296f7f4716b9eb5602aa3c842d (127.6.204.253:42483)
W20240502 08:02:31.823799 8053 leader_election.cc:334] T 00000000000000000000000000000000 P e80f89dd462b42ad8b210a98f0891253 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1f7d11296f7f4716b9eb5602aa3c842d (127.6.204.253:42483): Network error: Client connection negotiation failed: client connection to 127.6.204.253:42483: connect: Connection refused (error 111)
I20240502 08:02:31.824103 8577 raft_consensus.cc:482] T 00000000000000000000000000000000 P e80f89dd462b42ad8b210a98f0891253 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1f7d11296f7f4716b9eb5602aa3c842d)
I20240502 08:02:31.824196 8186 leader_election.cc:302] T 00000000000000000000000000000000 P 55808e5996304e98b57f3315454e4368 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55808e5996304e98b57f3315454e4368, e80f89dd462b42ad8b210a98f0891253; no voters:
I20240502 08:02:31.824676 8577 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e80f89dd462b42ad8b210a98f0891253 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:02:31.825197 8578 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 55808e5996304e98b57f3315454e4368 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:02:31.825646 8578 raft_consensus.cc:482] T 00000000000000000000000000000000 P 55808e5996304e98b57f3315454e4368 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1f7d11296f7f4716b9eb5602aa3c842d)
I20240502 08:02:31.826296 8578 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55808e5996304e98b57f3315454e4368 [term 1 FOLLOWER]: Advancing to term 2
W20240502 08:02:31.828590 8183 leader_election.cc:334] T 00000000000000000000000000000000 P 55808e5996304e98b57f3315454e4368 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1f7d11296f7f4716b9eb5602aa3c842d (127.6.204.253:42483): Network error: Client connection negotiation failed: client connection to 127.6.204.253:42483: connect: Connection refused (error 111)
I20240502 08:02:31.895401 8577 raft_consensus.cc:504] T 00000000000000000000000000000000 P e80f89dd462b42ad8b210a98f0891253 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e80f89dd462b42ad8b210a98f0891253" member_type: VOTER last_known_addr { host: "127.6.204.254" port: 33115 } } peers { permanent_uuid: "1f7d11296f7f4716b9eb5602aa3c842d" member_type: VOTER last_known_addr { host: "127.6.204.253" port: 42483 } } peers { permanent_uuid: "55808e5996304e98b57f3315454e4368" member_type: VOTER last_known_addr { host: "127.6.204.252" port: 46535 } }
I20240502 08:02:31.895401 8578 raft_consensus.cc:504] T 00000000000000000000000000000000 P 55808e5996304e98b57f3315454e4368 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e80f89dd462b42ad8b210a98f0891253" member_type: VOTER last_known_addr { host: "127.6.204.254" port: 33115 } } peers { permanent_uuid: "1f7d11296f7f4716b9eb5602aa3c842d" member_type: VOTER last_known_addr { host: "127.6.204.253" port: 42483 } } peers { permanent_uuid: "55808e5996304e98b57f3315454e4368" member_type: VOTER last_known_addr { host: "127.6.204.252" port: 46535 } }
I20240502 08:02:31.898900 8078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "55808e5996304e98b57f3315454e4368" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "e80f89dd462b42ad8b210a98f0891253"
I20240502 08:02:31.899780 8078 raft_consensus.cc:2379] T 00000000000000000000000000000000 P e80f89dd462b42ad8b210a98f0891253 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 55808e5996304e98b57f3315454e4368 in current term 2: Already voted for candidate e80f89dd462b42ad8b210a98f0891253 in this term.
I20240502 08:02:31.900220 8578 leader_election.cc:288] T 00000000000000000000000000000000 P 55808e5996304e98b57f3315454e4368 [CANDIDATE]: Term 2 election: Requested vote from peers e80f89dd462b42ad8b210a98f0891253 (127.6.204.254:33115), 1f7d11296f7f4716b9eb5602aa3c842d (127.6.204.253:42483)
I20240502 08:02:31.899812 8577 leader_election.cc:288] T 00000000000000000000000000000000 P e80f89dd462b42ad8b210a98f0891253 [CANDIDATE]: Term 2 election: Requested vote from peers 1f7d11296f7f4716b9eb5602aa3c842d (127.6.204.253:42483), 55808e5996304e98b57f3315454e4368 (127.6.204.252:46535)
I20240502 08:02:31.901394 8208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e80f89dd462b42ad8b210a98f0891253" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "55808e5996304e98b57f3315454e4368"
I20240502 08:02:31.902230 8208 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 55808e5996304e98b57f3315454e4368 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate e80f89dd462b42ad8b210a98f0891253 in current term 2: Already voted for candidate 55808e5996304e98b57f3315454e4368 in this term.
W20240502 08:02:31.902942 8183 leader_election.cc:334] T 00000000000000000000000000000000 P 55808e5996304e98b57f3315454e4368 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1f7d11296f7f4716b9eb5602aa3c842d (127.6.204.253:42483): Network error: Client connection negotiation failed: client connection to 127.6.204.253:42483: connect: Connection refused (error 111)
I20240502 08:02:31.903824 8183 leader_election.cc:302] T 00000000000000000000000000000000 P 55808e5996304e98b57f3315454e4368 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 55808e5996304e98b57f3315454e4368; no voters: 1f7d11296f7f4716b9eb5602aa3c842d, e80f89dd462b42ad8b210a98f0891253
W20240502 08:02:31.904031 8053 leader_election.cc:334] T 00000000000000000000000000000000 P e80f89dd462b42ad8b210a98f0891253 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1f7d11296f7f4716b9eb5602aa3c842d (127.6.204.253:42483): Network error: Client connection negotiation failed: client connection to 127.6.204.253:42483: connect: Connection refused (error 111)
I20240502 08:02:31.904407 8053 leader_election.cc:302] T 00000000000000000000000000000000 P e80f89dd462b42ad8b210a98f0891253 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e80f89dd462b42ad8b210a98f0891253; no voters: 1f7d11296f7f4716b9eb5602aa3c842d, 55808e5996304e98b57f3315454e4368
I20240502 08:02:31.904558 8578 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 55808e5996304e98b57f3315454e4368 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240502 08:02:31.905030 8577 raft_consensus.cc:2726] T 00000000000000000000000000000000 P e80f89dd462b42ad8b210a98f0891253 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240502 08:02:33.505610 8589 raft_consensus.cc:482] T 00000000000000000000000000000000 P 55808e5996304e98b57f3315454e4368 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:02:33.506062 8589 raft_consensus.cc:504] T 00000000000000000000000000000000 P 55808e5996304e98b57f3315454e4368 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e80f89dd462b42ad8b210a98f0891253" member_type: VOTER last_known_addr { host: "127.6.204.254" port: 33115 } } peers { permanent_uuid: "1f7d11296f7f4716b9eb5602aa3c842d" member_type: VOTER last_known_addr { host: "127.6.204.253" port: 42483 } } peers { permanent_uuid: "55808e5996304e98b57f3315454e4368" member_type: VOTER last_known_addr { host: "127.6.204.252" port: 46535 } }
I20240502 08:02:33.507680 8589 leader_election.cc:288] T 00000000000000000000000000000000 P 55808e5996304e98b57f3315454e4368 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e80f89dd462b42ad8b210a98f0891253 (127.6.204.254:33115), 1f7d11296f7f4716b9eb5602aa3c842d (127.6.204.253:42483)
I20240502 08:02:33.508603 8078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "55808e5996304e98b57f3315454e4368" candidate_term: 3 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "e80f89dd462b42ad8b210a98f0891253" is_pre_election: true
I20240502 08:02:33.509166 8078 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e80f89dd462b42ad8b210a98f0891253 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55808e5996304e98b57f3315454e4368 in term 2.
I20240502 08:02:33.510180 8186 leader_election.cc:302] T 00000000000000000000000000000000 P 55808e5996304e98b57f3315454e4368 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55808e5996304e98b57f3315454e4368, e80f89dd462b42ad8b210a98f0891253; no voters:
W20240502 08:02:33.510282 8183 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.204.253:42483: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240502 08:02:33.510885 8589 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 55808e5996304e98b57f3315454e4368 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 08:02:33.511160 8589 raft_consensus.cc:482] T 00000000000000000000000000000000 P 55808e5996304e98b57f3315454e4368 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:02:33.511440 8589 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55808e5996304e98b57f3315454e4368 [term 2 FOLLOWER]: Advancing to term 3
W20240502 08:02:33.513723 8183 leader_election.cc:334] T 00000000000000000000000000000000 P 55808e5996304e98b57f3315454e4368 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f7d11296f7f4716b9eb5602aa3c842d (127.6.204.253:42483): Network error: Client connection negotiation failed: client connection to 127.6.204.253:42483: connect: Connection refused (error 111)
I20240502 08:02:33.516228 8589 raft_consensus.cc:504] T 00000000000000000000000000000000 P 55808e5996304e98b57f3315454e4368 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e80f89dd462b42ad8b210a98f0891253" member_type: VOTER last_known_addr { host: "127.6.204.254" port: 33115 } } peers { permanent_uuid: "1f7d11296f7f4716b9eb5602aa3c842d" member_type: VOTER last_known_addr { host: "127.6.204.253" port: 42483 } } peers { permanent_uuid: "55808e5996304e98b57f3315454e4368" member_type: VOTER last_known_addr { host: "127.6.204.252" port: 46535 } }
I20240502 08:02:33.517750 8589 leader_election.cc:288] T 00000000000000000000000000000000 P 55808e5996304e98b57f3315454e4368 [CANDIDATE]: Term 3 election: Requested vote from peers e80f89dd462b42ad8b210a98f0891253 (127.6.204.254:33115), 1f7d11296f7f4716b9eb5602aa3c842d (127.6.204.253:42483)
I20240502 08:02:33.518646 8078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "55808e5996304e98b57f3315454e4368" candidate_term: 3 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "e80f89dd462b42ad8b210a98f0891253"
I20240502 08:02:33.519325 8078 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e80f89dd462b42ad8b210a98f0891253 [term 2 FOLLOWER]: Advancing to term 3
W20240502 08:02:33.521911 8183 leader_election.cc:334] T 00000000000000000000000000000000 P 55808e5996304e98b57f3315454e4368 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 1f7d11296f7f4716b9eb5602aa3c842d (127.6.204.253:42483): Network error: Client connection negotiation failed: client connection to 127.6.204.253:42483: connect: Connection refused (error 111)
I20240502 08:02:33.524787 8078 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e80f89dd462b42ad8b210a98f0891253 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55808e5996304e98b57f3315454e4368 in term 3.
I20240502 08:02:33.525709 8186 leader_election.cc:302] T 00000000000000000000000000000000 P 55808e5996304e98b57f3315454e4368 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 55808e5996304e98b57f3315454e4368, e80f89dd462b42ad8b210a98f0891253; no voters: 1f7d11296f7f4716b9eb5602aa3c842d
I20240502 08:02:33.526453 8589 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 55808e5996304e98b57f3315454e4368 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 08:02:33.527586 8589 raft_consensus.cc:686] T 00000000000000000000000000000000 P 55808e5996304e98b57f3315454e4368 [term 3 LEADER]: Becoming Leader. State: Replica: 55808e5996304e98b57f3315454e4368, State: Running, Role: LEADER
I20240502 08:02:33.528237 8589 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55808e5996304e98b57f3315454e4368 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e80f89dd462b42ad8b210a98f0891253" member_type: VOTER last_known_addr { host: "127.6.204.254" port: 33115 } } peers { permanent_uuid: "1f7d11296f7f4716b9eb5602aa3c842d" member_type: VOTER last_known_addr { host: "127.6.204.253" port: 42483 } } peers { permanent_uuid: "55808e5996304e98b57f3315454e4368" member_type: VOTER last_known_addr { host: "127.6.204.252" port: 46535 } }
I20240502 08:02:33.532069 8592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55808e5996304e98b57f3315454e4368 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55808e5996304e98b57f3315454e4368. Latest consensus state: current_term: 3 leader_uuid: "55808e5996304e98b57f3315454e4368" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e80f89dd462b42ad8b210a98f0891253" member_type: VOTER last_known_addr { host: "127.6.204.254" port: 33115 } } peers { permanent_uuid: "1f7d11296f7f4716b9eb5602aa3c842d" member_type: VOTER last_known_addr { host: "127.6.204.253" port: 42483 } } peers { permanent_uuid: "55808e5996304e98b57f3315454e4368" member_type: VOTER last_known_addr { host: "127.6.204.252" port: 46535 } } }
I20240502 08:02:33.532624 8592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55808e5996304e98b57f3315454e4368 [sys.catalog]: This master's current role is: LEADER
I20240502 08:02:33.533998 8594 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 08:02:33.539207 8594 catalog_manager.cc:669] Loaded metadata for table test-workload [id=60408b8ba1c74955a5c3bade6d8e423d]
I20240502 08:02:33.544166 8594 tablet_loader.cc:96] loaded metadata for tablet 42c8e173d0da466da778ec873c9dc745 (table test-workload [id=60408b8ba1c74955a5c3bade6d8e423d])
I20240502 08:02:33.545090 8594 tablet_loader.cc:96] loaded metadata for tablet 48160b0a72244bbda66c5a5f927379d3 (table test-workload [id=60408b8ba1c74955a5c3bade6d8e423d])
I20240502 08:02:33.545964 8594 tablet_loader.cc:96] loaded metadata for tablet 5f82424c85be4acab0d3a5bc3ed841c8 (table test-workload [id=60408b8ba1c74955a5c3bade6d8e423d])
I20240502 08:02:33.546837 8594 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 08:02:33.549758 8594 catalog_manager.cc:1259] Loaded cluster ID: 9691e79e85f64e07b05f9266f9654bb6
I20240502 08:02:33.550088 8594 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 08:02:33.557001 8594 catalog_manager.cc:1504] Loading token signing keys...
I20240502 08:02:33.560312 8594 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 55808e5996304e98b57f3315454e4368: Loaded TSK: 0
I20240502 08:02:33.561301 8594 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 08:02:33.991365 8275 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:02:33.992053 8275 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:02:34.056690 8078 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e80f89dd462b42ad8b210a98f0891253 [term 3 FOLLOWER]: Refusing update from remote peer 55808e5996304e98b57f3315454e4368: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20240502 08:02:34.059008 8596 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 55808e5996304e98b57f3315454e4368 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e80f89dd462b42ad8b210a98f0891253" member_type: VOTER last_known_addr { host: "127.6.204.254" port: 33115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
W20240502 08:02:34.066120 8183 consensus_peers.cc:485] T 00000000000000000000000000000000 P 55808e5996304e98b57f3315454e4368 -> Peer 1f7d11296f7f4716b9eb5602aa3c842d (127.6.204.253:42483): Couldn't send request to peer 1f7d11296f7f4716b9eb5602aa3c842d. Status: Network error: Client connection negotiation failed: client connection to 127.6.204.253:42483: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:02:34.068728 8600 sys_catalog.cc:455] T 00000000000000000000000000000000 P e80f89dd462b42ad8b210a98f0891253 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55808e5996304e98b57f3315454e4368. Latest consensus state: current_term: 3 leader_uuid: "55808e5996304e98b57f3315454e4368" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e80f89dd462b42ad8b210a98f0891253" member_type: VOTER last_known_addr { host: "127.6.204.254" port: 33115 } } peers { permanent_uuid: "1f7d11296f7f4716b9eb5602aa3c842d" member_type: VOTER last_known_addr { host: "127.6.204.253" port: 42483 } } peers { permanent_uuid: "55808e5996304e98b57f3315454e4368" member_type: VOTER last_known_addr { host: "127.6.204.252" port: 46535 } } }
I20240502 08:02:34.069643 8600 sys_catalog.cc:458] T 00000000000000000000000000000000 P e80f89dd462b42ad8b210a98f0891253 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 08:02:34.072438 8597 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55808e5996304e98b57f3315454e4368 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "55808e5996304e98b57f3315454e4368" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e80f89dd462b42ad8b210a98f0891253" member_type: VOTER last_known_addr { host: "127.6.204.254" port: 33115 } } peers { permanent_uuid: "1f7d11296f7f4716b9eb5602aa3c842d" member_type: VOTER last_known_addr { host: "127.6.204.253" port: 42483 } } peers { permanent_uuid: "55808e5996304e98b57f3315454e4368" member_type: VOTER last_known_addr { host: "127.6.204.252" port: 46535 } } }
I20240502 08:02:34.073179 8597 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55808e5996304e98b57f3315454e4368 [sys.catalog]: This master's current role is: LEADER
I20240502 08:02:34.074427 8600 sys_catalog.cc:455] T 00000000000000000000000000000000 P e80f89dd462b42ad8b210a98f0891253 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "55808e5996304e98b57f3315454e4368" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e80f89dd462b42ad8b210a98f0891253" member_type: VOTER last_known_addr { host: "127.6.204.254" port: 33115 } } peers { permanent_uuid: "1f7d11296f7f4716b9eb5602aa3c842d" member_type: VOTER last_known_addr { host: "127.6.204.253" port: 42483 } } peers { permanent_uuid: "55808e5996304e98b57f3315454e4368" member_type: VOTER last_known_addr { host: "127.6.204.252" port: 46535 } } }
I20240502 08:02:34.075246 8600 sys_catalog.cc:458] T 00000000000000000000000000000000 P e80f89dd462b42ad8b210a98f0891253 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 08:02:34.155711 6963 tablet_server.cc:178] TabletServer@127.6.204.193:0 shutting down...
I20240502 08:02:34.183084 6963 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 08:02:34.183796 6963 tablet_replica.cc:330] T 42c8e173d0da466da778ec873c9dc745 P b5c5a66357ed43c4973a42e64caa8446: stopping tablet replica
I20240502 08:02:34.184425 6963 raft_consensus.cc:2229] T 42c8e173d0da466da778ec873c9dc745 P b5c5a66357ed43c4973a42e64caa8446 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:02:34.185001 6963 raft_consensus.cc:2258] T 42c8e173d0da466da778ec873c9dc745 P b5c5a66357ed43c4973a42e64caa8446 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:02:34.204860 6963 tablet_server.cc:194] TabletServer@127.6.204.193:0 shutdown complete.
I20240502 08:02:34.215785 6963 tablet_server.cc:178] TabletServer@127.6.204.194:0 shutting down...
I20240502 08:02:34.238663 6963 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 08:02:34.239382 6963 tablet_replica.cc:330] T 5f82424c85be4acab0d3a5bc3ed841c8 P d35dbf5f483f45199c94cf946704a35a: stopping tablet replica
I20240502 08:02:34.239925 6963 raft_consensus.cc:2229] T 5f82424c85be4acab0d3a5bc3ed841c8 P d35dbf5f483f45199c94cf946704a35a [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:02:34.240486 6963 raft_consensus.cc:2258] T 5f82424c85be4acab0d3a5bc3ed841c8 P d35dbf5f483f45199c94cf946704a35a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:02:34.260289 6963 tablet_server.cc:194] TabletServer@127.6.204.194:0 shutdown complete.
I20240502 08:02:34.270841 6963 tablet_server.cc:178] TabletServer@127.6.204.195:0 shutting down...
I20240502 08:02:34.300796 8517 heartbeater.cc:503] Master 127.6.204.252:46535 was elected leader, sending a full tablet report...
I20240502 08:02:34.310415 6963 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 08:02:34.311048 6963 tablet_replica.cc:330] T 48160b0a72244bbda66c5a5f927379d3 P ecf2a63dfa304fe48ecc58b49ce204ff: stopping tablet replica
I20240502 08:02:34.311517 6963 raft_consensus.cc:2229] T 48160b0a72244bbda66c5a5f927379d3 P ecf2a63dfa304fe48ecc58b49ce204ff [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:02:34.311914 6963 raft_consensus.cc:2258] T 48160b0a72244bbda66c5a5f927379d3 P ecf2a63dfa304fe48ecc58b49ce204ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:02:34.330267 6963 tablet_server.cc:194] TabletServer@127.6.204.195:0 shutdown complete.
I20240502 08:02:34.339939 6963 master.cc:537] Master@127.6.204.254:33115 shutting down...
I20240502 08:02:34.362634 6963 raft_consensus.cc:2229] T 00000000000000000000000000000000 P e80f89dd462b42ad8b210a98f0891253 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 08:02:34.363214 6963 raft_consensus.cc:2258] T 00000000000000000000000000000000 P e80f89dd462b42ad8b210a98f0891253 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 08:02:34.363539 6963 tablet_replica.cc:330] T 00000000000000000000000000000000 P e80f89dd462b42ad8b210a98f0891253: stopping tablet replica
I20240502 08:02:34.382566 6963 master.cc:552] Master@127.6.204.254:33115 shutdown complete.
I20240502 08:02:34.395606 6963 master.cc:537] Master@127.6.204.252:46535 shutting down...
I20240502 08:02:34.411101 6963 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 55808e5996304e98b57f3315454e4368 [term 3 LEADER]: Raft consensus shutting down.
I20240502 08:02:34.412067 6963 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 55808e5996304e98b57f3315454e4368 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 08:02:34.412489 6963 tablet_replica.cc:330] T 00000000000000000000000000000000 P 55808e5996304e98b57f3315454e4368: stopping tablet replica
I20240502 08:02:34.431810 6963 master.cc:552] Master@127.6.204.252:46535 shutdown complete.
[ OK ] AutoRebalancerTest.NextLeaderResumesAutoRebalancing (6829 ms)
[ RUN ] AutoRebalancerTest.MovesScheduledIfAddTserver
I20240502 08:02:34.463276 6963 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.6.204.254:36313
I20240502 08:02:34.464345 6963 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:02:34.470475 8606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:02:34.470551 8607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:02:34.472011 8609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:02:34.472169 6963 server_base.cc:1034] running on GCE node
I20240502 08:02:34.473047 6963 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:02:34.473227 6963 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:02:34.473377 6963 hybrid_clock.cc:648] HybridClock initialized: now 1714636954473364 us; error 0 us; skew 500 ppm
I20240502 08:02:34.473893 6963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:02:34.476375 6963 webserver.cc:460] Webserver started at http://127.6.204.254:33619/ using document root <none> and password file <none>
I20240502 08:02:34.476841 6963 fs_manager.cc:359] Metadata directory not provided
I20240502 08:02:34.477005 6963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:02:34.477278 6963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:02:34.478636 6963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714636933625983-6963-0/minicluster-data/master-0-root/instance:
uuid: "4479d3bf0da04a05a10cf6d93d86ef40"
format_stamp: "Formatted at 2024-05-02 08:02:34 on dist-test-slave-32xs"
I20240502 08:02:34.483110 6963 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240502 08:02:34.486490 8614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:02:34.487205 6963 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20240502 08:02:34.487463 6963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714636933625983-6963-0/minicluster-data/master-0-root
uuid: "4479d3bf0da04a05a10cf6d93d86ef40"
format_stamp: "Formatted at 2024-05-02 08:02:34 on dist-test-slave-32xs"
I20240502 08:02:34.487736 6963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714636933625983-6963-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714636933625983-6963-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714636933625983-6963-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:02:34.503588 6963 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:02:34.504843 6963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:02:34.542371 6963 rpc_server.cc:307] RPC server started. Bound to: 127.6.204.254:36313
I20240502 08:02:34.542466 8665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.204.254:36313 every 8 connection(s)
I20240502 08:02:34.546237 8666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:02:34.556355 8666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4479d3bf0da04a05a10cf6d93d86ef40: Bootstrap starting.
I20240502 08:02:34.560592 8666 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4479d3bf0da04a05a10cf6d93d86ef40: Neither blocks nor log segments found. Creating new log.
I20240502 08:02:34.564755 8666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4479d3bf0da04a05a10cf6d93d86ef40: No bootstrap required, opened a new log
I20240502 08:02:34.566716 8666 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4479d3bf0da04a05a10cf6d93d86ef40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4479d3bf0da04a05a10cf6d93d86ef40" member_type: VOTER }
I20240502 08:02:34.567088 8666 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4479d3bf0da04a05a10cf6d93d86ef40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:02:34.567323 8666 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4479d3bf0da04a05a10cf6d93d86ef40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4479d3bf0da04a05a10cf6d93d86ef40, State: Initialized, Role: FOLLOWER
I20240502 08:02:34.567893 8666 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4479d3bf0da04a05a10cf6d93d86ef40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4479d3bf0da04a05a10cf6d93d86ef40" member_type: VOTER }
I20240502 08:02:34.568344 8666 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4479d3bf0da04a05a10cf6d93d86ef40 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:02:34.568550 8666 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4479d3bf0da04a05a10cf6d93d86ef40 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:02:34.568816 8666 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4479d3bf0da04a05a10cf6d93d86ef40 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:02:34.572984 8666 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4479d3bf0da04a05a10cf6d93d86ef40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4479d3bf0da04a05a10cf6d93d86ef40" member_type: VOTER }
I20240502 08:02:34.573486 8666 leader_election.cc:302] T 00000000000000000000000000000000 P 4479d3bf0da04a05a10cf6d93d86ef40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4479d3bf0da04a05a10cf6d93d86ef40; no voters:
I20240502 08:02:34.574609 8666 leader_election.cc:288] T 00000000000000000000000000000000 P 4479d3bf0da04a05a10cf6d93d86ef40 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:02:34.574929 8669 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4479d3bf0da04a05a10cf6d93d86ef40 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:02:34.576259 8669 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4479d3bf0da04a05a10cf6d93d86ef40 [term 1 LEADER]: Becoming Leader. State: Replica: 4479d3bf0da04a05a10cf6d93d86ef40, State: Running, Role: LEADER
I20240502 08:02:34.576977 8669 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4479d3bf0da04a05a10cf6d93d86ef40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4479d3bf0da04a05a10cf6d93d86ef40" member_type: VOTER }
I20240502 08:02:34.577468 8666 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4479d3bf0da04a05a10cf6d93d86ef40 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 08:02:34.579810 8671 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4479d3bf0da04a05a10cf6d93d86ef40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4479d3bf0da04a05a10cf6d93d86ef40. Latest consensus state: current_term: 1 leader_uuid: "4479d3bf0da04a05a10cf6d93d86ef40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4479d3bf0da04a05a10cf6d93d86ef40" member_type: VOTER } }
I20240502 08:02:34.579891 8670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4479d3bf0da04a05a10cf6d93d86ef40 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4479d3bf0da04a05a10cf6d93d86ef40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4479d3bf0da04a05a10cf6d93d86ef40" member_type: VOTER } }
I20240502 08:02:34.580488 8671 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4479d3bf0da04a05a10cf6d93d86ef40 [sys.catalog]: This master's current role is: LEADER
I20240502 08:02:34.580646 8670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4479d3bf0da04a05a10cf6d93d86ef40 [sys.catalog]: This master's current role is: LEADER
I20240502 08:02:34.584225 8674 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 08:02:34.590557 8674 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 08:02:34.592001 6963 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 08:02:34.599387 8674 catalog_manager.cc:1347] Generated new cluster ID: 0d17144cc45f452598c21c19f6382b88
I20240502 08:02:34.599695 8674 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 08:02:34.625787 8674 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 08:02:34.627082 8674 catalog_manager.cc:1504] Loading token signing keys...
I20240502 08:02:34.638490 8674 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 4479d3bf0da04a05a10cf6d93d86ef40: Generated new TSK 0
I20240502 08:02:34.639145 8674 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 08:02:34.658535 6963 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:02:34.664503 8687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:02:34.665992 8688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:02:34.667513 6963 server_base.cc:1034] running on GCE node
W20240502 08:02:34.667649 8690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:02:34.668506 6963 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:02:34.668692 6963 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:02:34.668869 6963 hybrid_clock.cc:648] HybridClock initialized: now 1714636954668849 us; error 0 us; skew 500 ppm
I20240502 08:02:34.669381 6963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:02:34.671849 6963 webserver.cc:460] Webserver started at http://127.6.204.193:46519/ using document root <none> and password file <none>
I20240502 08:02:34.672327 6963 fs_manager.cc:359] Metadata directory not provided
I20240502 08:02:34.672510 6963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:02:34.672785 6963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:02:34.673867 6963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714636933625983-6963-0/minicluster-data/ts-0-root/instance:
uuid: "ab8a50f16fae448da57e40b4b78bfb03"
format_stamp: "Formatted at 2024-05-02 08:02:34 on dist-test-slave-32xs"
I20240502 08:02:34.678385 6963 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240502 08:02:34.681718 8695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:02:34.682504 6963 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 08:02:34.682787 6963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714636933625983-6963-0/minicluster-data/ts-0-root
uuid: "ab8a50f16fae448da57e40b4b78bfb03"
format_stamp: "Formatted at 2024-05-02 08:02:34 on dist-test-slave-32xs"
I20240502 08:02:34.683065 6963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714636933625983-6963-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714636933625983-6963-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714636933625983-6963-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:02:34.707823 6963 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:02:34.708973 6963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:02:34.710456 6963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:02:34.712792 6963 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:02:34.712992 6963 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:02:34.713217 6963 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:02:34.713379 6963 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:02:34.752532 6963 rpc_server.cc:307] RPC server started. Bound to: 127.6.204.193:42649
I20240502 08:02:34.752635 8757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.204.193:42649 every 8 connection(s)
I20240502 08:02:34.757091 6963 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:02:34.763929 8762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:02:34.764647 8763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:02:34.768713 6963 server_base.cc:1034] running on GCE node
W20240502 08:02:34.769145 8765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:02:34.770166 6963 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:02:34.770443 6963 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:02:34.770606 6963 hybrid_clock.cc:648] HybridClock initialized: now 1714636954770590 us; error 0 us; skew 500 ppm
I20240502 08:02:34.771121 6963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:02:34.771332 8758 heartbeater.cc:347] Connected to a master server at 127.6.204.254:36313
I20240502 08:02:34.771669 8758 heartbeater.cc:464] Registering TS with master...
I20240502 08:02:34.772424 8758 heartbeater.cc:511] Master 127.6.204.254:36313 requested a full tablet report, sending...
I20240502 08:02:34.773600 6963 webserver.cc:460] Webserver started at http://127.6.204.194:36585/ using document root <none> and password file <none>
I20240502 08:02:34.774080 6963 fs_manager.cc:359] Metadata directory not provided
I20240502 08:02:34.774302 6963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:02:34.774559 6963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:02:34.774578 8631 ts_manager.cc:194] Registered new tserver with Master: ab8a50f16fae448da57e40b4b78bfb03 (127.6.204.193:42649)
I20240502 08:02:34.775784 6963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714636933625983-6963-0/minicluster-data/ts-1-root/instance:
uuid: "e7b4ca4d63bd42708824172a58c462a2"
format_stamp: "Formatted at 2024-05-02 08:02:34 on dist-test-slave-32xs"
I20240502 08:02:34.776449 8631 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34384
I20240502 08:02:34.780396 6963 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 08:02:34.783840 8770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:02:34.784679 6963 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 08:02:34.784963 6963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714636933625983-6963-0/minicluster-data/ts-1-root
uuid: "e7b4ca4d63bd42708824172a58c462a2"
format_stamp: "Formatted at 2024-05-02 08:02:34 on dist-test-slave-32xs"
I20240502 08:02:34.785210 6963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714636933625983-6963-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714636933625983-6963-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714636933625983-6963-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:02:34.794391 6963 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:02:34.795475 6963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:02:34.796809 6963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:02:34.799041 6963 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:02:34.799213 6963 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:02:34.799441 6963 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:02:34.799580 6963 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:02:34.838049 6963 rpc_server.cc:307] RPC server started. Bound to: 127.6.204.194:41577
I20240502 08:02:34.838238 8832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.204.194:41577 every 8 connection(s)
I20240502 08:02:34.842805 6963 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:02:34.849689 8836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:02:34.851754 8837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:02:34.853891 8839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:02:34.854684 6963 server_base.cc:1034] running on GCE node
I20240502 08:02:34.855595 6963 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20240502 08:02:34.855707 8833 heartbeater.cc:347] Connected to a master server at 127.6.204.254:36313
W20240502 08:02:34.855880 6963 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:02:34.856107 6963 hybrid_clock.cc:648] HybridClock initialized: now 1714636954856085 us; error 0 us; skew 500 ppm
I20240502 08:02:34.856108 8833 heartbeater.cc:464] Registering TS with master...
I20240502 08:02:34.856990 6963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:02:34.857218 8833 heartbeater.cc:511] Master 127.6.204.254:36313 requested a full tablet report, sending...
I20240502 08:02:34.859496 8631 ts_manager.cc:194] Registered new tserver with Master: e7b4ca4d63bd42708824172a58c462a2 (127.6.204.194:41577)
I20240502 08:02:34.860039 6963 webserver.cc:460] Webserver started at http://127.6.204.195:35853/ using document root <none> and password file <none>
I20240502 08:02:34.860508 6963 fs_manager.cc:359] Metadata directory not provided
I20240502 08:02:34.860715 6963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:02:34.860976 6963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:02:34.861147 8631 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34400
I20240502 08:02:34.862510 6963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714636933625983-6963-0/minicluster-data/ts-2-root/instance:
uuid: "44783b6715ac408982906300539ae6a0"
format_stamp: "Formatted at 2024-05-02 08:02:34 on dist-test-slave-32xs"
I20240502 08:02:34.867172 6963 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20240502 08:02:34.870370 8844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:02:34.871124 6963 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 08:02:34.871397 6963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714636933625983-6963-0/minicluster-data/ts-2-root
uuid: "44783b6715ac408982906300539ae6a0"
format_stamp: "Formatted at 2024-05-02 08:02:34 on dist-test-slave-32xs"
I20240502 08:02:34.871680 6963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714636933625983-6963-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714636933625983-6963-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714636933625983-6963-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:02:34.880851 6963 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:02:34.882038 6963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:02:34.883543 6963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:02:34.886015 6963 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:02:34.886296 6963 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20240502 08:02:34.886516 6963 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:02:34.886699 6963 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:02:34.927769 6963 rpc_server.cc:307] RPC server started. Bound to: 127.6.204.195:38657
I20240502 08:02:34.927850 8906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.204.195:38657 every 8 connection(s)
I20240502 08:02:34.941207 8907 heartbeater.cc:347] Connected to a master server at 127.6.204.254:36313
I20240502 08:02:34.941547 8907 heartbeater.cc:464] Registering TS with master...
I20240502 08:02:34.942246 8907 heartbeater.cc:511] Master 127.6.204.254:36313 requested a full tablet report, sending...
I20240502 08:02:34.944109 8631 ts_manager.cc:194] Registered new tserver with Master: 44783b6715ac408982906300539ae6a0 (127.6.204.195:38657)
I20240502 08:02:34.944695 6963 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013704611s
I20240502 08:02:34.945447 8631 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34412
I20240502 08:02:35.779024 8758 heartbeater.cc:503] Master 127.6.204.254:36313 was elected leader, sending a full tablet report...
I20240502 08:02:35.809491 6963 test_util.cc:268] Using random seed: 111900347
I20240502 08:02:35.833055 8631 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34414:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377?""\004\001\000\377\377\377?"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:02:35.835438 8631 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:02:35.864353 8833 heartbeater.cc:503] Master 127.6.204.254:36313 was elected leader, sending a full tablet report...
I20240502 08:02:35.891369 8723 tablet_service.cc:1467] Processing CreateTablet for tablet 8ad9d4c321a449539ca014fc4e89005d (DEFAULT_TABLE table=test-workload [id=6f7ba7649f0648ff9867b9a0d015ff38]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240502 08:02:35.891738 8722 tablet_service.cc:1467] Processing CreateTablet for tablet a3b92bb4dcb7446490472014a72828dc (DEFAULT_TABLE table=test-workload [id=6f7ba7649f0648ff9867b9a0d015ff38]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240502 08:02:35.892870 8723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ad9d4c321a449539ca014fc4e89005d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:02:35.891738 8798 tablet_service.cc:1467] Processing CreateTablet for tablet 8ad9d4c321a449539ca014fc4e89005d (DEFAULT_TABLE table=test-workload [id=6f7ba7649f0648ff9867b9a0d015ff38]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240502 08:02:35.893618 8722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3b92bb4dcb7446490472014a72828dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:02:35.894285 8798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ad9d4c321a449539ca014fc4e89005d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:02:35.897754 8797 tablet_service.cc:1467] Processing CreateTablet for tablet a3b92bb4dcb7446490472014a72828dc (DEFAULT_TABLE table=test-workload [id=6f7ba7649f0648ff9867b9a0d015ff38]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240502 08:02:35.899060 8797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3b92bb4dcb7446490472014a72828dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:02:35.905242 8872 tablet_service.cc:1467] Processing CreateTablet for tablet 8ad9d4c321a449539ca014fc4e89005d (DEFAULT_TABLE table=test-workload [id=6f7ba7649f0648ff9867b9a0d015ff38]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240502 08:02:35.905689 8871 tablet_service.cc:1467] Processing CreateTablet for tablet a3b92bb4dcb7446490472014a72828dc (DEFAULT_TABLE table=test-workload [id=6f7ba7649f0648ff9867b9a0d015ff38]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240502 08:02:35.906602 8872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ad9d4c321a449539ca014fc4e89005d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:02:35.907923 8871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3b92bb4dcb7446490472014a72828dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:02:35.927407 8927 tablet_bootstrap.cc:492] T a3b92bb4dcb7446490472014a72828dc P ab8a50f16fae448da57e40b4b78bfb03: Bootstrap starting.
I20240502 08:02:35.930008 8928 tablet_bootstrap.cc:492] T a3b92bb4dcb7446490472014a72828dc P e7b4ca4d63bd42708824172a58c462a2: Bootstrap starting.
I20240502 08:02:35.935539 8929 tablet_bootstrap.cc:492] T 8ad9d4c321a449539ca014fc4e89005d P 44783b6715ac408982906300539ae6a0: Bootstrap starting.
I20240502 08:02:35.937341 8927 tablet_bootstrap.cc:654] T a3b92bb4dcb7446490472014a72828dc P ab8a50f16fae448da57e40b4b78bfb03: Neither blocks nor log segments found. Creating new log.
I20240502 08:02:35.938460 8928 tablet_bootstrap.cc:654] T a3b92bb4dcb7446490472014a72828dc P e7b4ca4d63bd42708824172a58c462a2: Neither blocks nor log segments found. Creating new log.
I20240502 08:02:35.941466 8929 tablet_bootstrap.cc:654] T 8ad9d4c321a449539ca014fc4e89005d P 44783b6715ac408982906300539ae6a0: Neither blocks nor log segments found. Creating new log.
I20240502 08:02:35.944061 8928 tablet_bootstrap.cc:492] T a3b92bb4dcb7446490472014a72828dc P e7b4ca4d63bd42708824172a58c462a2: No bootstrap required, opened a new log
I20240502 08:02:35.944626 8928 ts_tablet_manager.cc:1367] T a3b92bb4dcb7446490472014a72828dc P e7b4ca4d63bd42708824172a58c462a2: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20240502 08:02:35.947129 8927 tablet_bootstrap.cc:492] T a3b92bb4dcb7446490472014a72828dc P ab8a50f16fae448da57e40b4b78bfb03: No bootstrap required, opened a new log
I20240502 08:02:35.947000 8928 raft_consensus.cc:348] T a3b92bb4dcb7446490472014a72828dc P e7b4ca4d63bd42708824172a58c462a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab8a50f16fae448da57e40b4b78bfb03" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42649 } } peers { permanent_uuid: "e7b4ca4d63bd42708824172a58c462a2" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 41577 } } peers { permanent_uuid: "44783b6715ac408982906300539ae6a0" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38657 } }
I20240502 08:02:35.947804 8928 raft_consensus.cc:374] T a3b92bb4dcb7446490472014a72828dc P e7b4ca4d63bd42708824172a58c462a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:02:35.947795 8927 ts_tablet_manager.cc:1367] T a3b92bb4dcb7446490472014a72828dc P ab8a50f16fae448da57e40b4b78bfb03: Time spent bootstrapping tablet: real 0.021s user 0.010s sys 0.002s
I20240502 08:02:35.948175 8928 raft_consensus.cc:729] T a3b92bb4dcb7446490472014a72828dc P e7b4ca4d63bd42708824172a58c462a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7b4ca4d63bd42708824172a58c462a2, State: Initialized, Role: FOLLOWER
I20240502 08:02:35.948643 8907 heartbeater.cc:503] Master 127.6.204.254:36313 was elected leader, sending a full tablet report...
I20240502 08:02:35.949100 8928 consensus_queue.cc:260] T a3b92bb4dcb7446490472014a72828dc P e7b4ca4d63bd42708824172a58c462a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab8a50f16fae448da57e40b4b78bfb03" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42649 } } peers { permanent_uuid: "e7b4ca4d63bd42708824172a58c462a2" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 41577 } } peers { permanent_uuid: "44783b6715ac408982906300539ae6a0" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38657 } }
I20240502 08:02:35.951685 8927 raft_consensus.cc:348] T a3b92bb4dcb7446490472014a72828dc P ab8a50f16fae448da57e40b4b78bfb03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab8a50f16fae448da57e40b4b78bfb03" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42649 } } peers { permanent_uuid: "e7b4ca4d63bd42708824172a58c462a2" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 41577 } } peers { permanent_uuid: "44783b6715ac408982906300539ae6a0" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38657 } }
I20240502 08:02:35.952726 8927 raft_consensus.cc:374] T a3b92bb4dcb7446490472014a72828dc P ab8a50f16fae448da57e40b4b78bfb03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:02:35.953096 8927 raft_consensus.cc:729] T a3b92bb4dcb7446490472014a72828dc P ab8a50f16fae448da57e40b4b78bfb03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab8a50f16fae448da57e40b4b78bfb03, State: Initialized, Role: FOLLOWER
I20240502 08:02:35.954358 8927 consensus_queue.cc:260] T a3b92bb4dcb7446490472014a72828dc P ab8a50f16fae448da57e40b4b78bfb03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab8a50f16fae448da57e40b4b78bfb03" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42649 } } peers { permanent_uuid: "e7b4ca4d63bd42708824172a58c462a2" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 41577 } } peers { permanent_uuid: "44783b6715ac408982906300539ae6a0" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38657 } }
I20240502 08:02:35.958710 8928 ts_tablet_manager.cc:1398] T a3b92bb4dcb7446490472014a72828dc P e7b4ca4d63bd42708824172a58c462a2: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20240502 08:02:35.959812 8928 tablet_bootstrap.cc:492] T 8ad9d4c321a449539ca014fc4e89005d P e7b4ca4d63bd42708824172a58c462a2: Bootstrap starting.
I20240502 08:02:35.964408 8927 ts_tablet_manager.cc:1398] T a3b92bb4dcb7446490472014a72828dc P ab8a50f16fae448da57e40b4b78bfb03: Time spent starting tablet: real 0.016s user 0.010s sys 0.004s
I20240502 08:02:35.965755 8928 tablet_bootstrap.cc:654] T 8ad9d4c321a449539ca014fc4e89005d P e7b4ca4d63bd42708824172a58c462a2: Neither blocks nor log segments found. Creating new log.
I20240502 08:02:35.967257 8929 tablet_bootstrap.cc:492] T 8ad9d4c321a449539ca014fc4e89005d P 44783b6715ac408982906300539ae6a0: No bootstrap required, opened a new log
I20240502 08:02:35.967446 8927 tablet_bootstrap.cc:492] T 8ad9d4c321a449539ca014fc4e89005d P ab8a50f16fae448da57e40b4b78bfb03: Bootstrap starting.
I20240502 08:02:35.967774 8929 ts_tablet_manager.cc:1367] T 8ad9d4c321a449539ca014fc4e89005d P 44783b6715ac408982906300539ae6a0: Time spent bootstrapping tablet: real 0.033s user 0.006s sys 0.017s
I20240502 08:02:35.974014 8927 tablet_bootstrap.cc:654] T 8ad9d4c321a449539ca014fc4e89005d P ab8a50f16fae448da57e40b4b78bfb03: Neither blocks nor log segments found. Creating new log.
I20240502 08:02:35.975546 8929 raft_consensus.cc:348] T 8ad9d4c321a449539ca014fc4e89005d P 44783b6715ac408982906300539ae6a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7b4ca4d63bd42708824172a58c462a2" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 41577 } } peers { permanent_uuid: "44783b6715ac408982906300539ae6a0" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38657 } } peers { permanent_uuid: "ab8a50f16fae448da57e40b4b78bfb03" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42649 } }
I20240502 08:02:35.976744 8929 raft_consensus.cc:374] T 8ad9d4c321a449539ca014fc4e89005d P 44783b6715ac408982906300539ae6a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:02:35.977188 8929 raft_consensus.cc:729] T 8ad9d4c321a449539ca014fc4e89005d P 44783b6715ac408982906300539ae6a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44783b6715ac408982906300539ae6a0, State: Initialized, Role: FOLLOWER
I20240502 08:02:35.978214 8929 consensus_queue.cc:260] T 8ad9d4c321a449539ca014fc4e89005d P 44783b6715ac408982906300539ae6a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7b4ca4d63bd42708824172a58c462a2" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 41577 } } peers { permanent_uuid: "44783b6715ac408982906300539ae6a0" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38657 } } peers { permanent_uuid: "ab8a50f16fae448da57e40b4b78bfb03" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42649 } }
I20240502 08:02:35.986219 8928 tablet_bootstrap.cc:492] T 8ad9d4c321a449539ca014fc4e89005d P e7b4ca4d63bd42708824172a58c462a2: No bootstrap required, opened a new log
I20240502 08:02:35.986867 8928 ts_tablet_manager.cc:1367] T 8ad9d4c321a449539ca014fc4e89005d P e7b4ca4d63bd42708824172a58c462a2: Time spent bootstrapping tablet: real 0.027s user 0.014s sys 0.011s
I20240502 08:02:35.988101 8929 ts_tablet_manager.cc:1398] T 8ad9d4c321a449539ca014fc4e89005d P 44783b6715ac408982906300539ae6a0: Time spent starting tablet: real 0.020s user 0.008s sys 0.005s
I20240502 08:02:35.989595 8929 tablet_bootstrap.cc:492] T a3b92bb4dcb7446490472014a72828dc P 44783b6715ac408982906300539ae6a0: Bootstrap starting.
I20240502 08:02:35.989140 8928 raft_consensus.cc:348] T 8ad9d4c321a449539ca014fc4e89005d P e7b4ca4d63bd42708824172a58c462a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7b4ca4d63bd42708824172a58c462a2" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 41577 } } peers { permanent_uuid: "44783b6715ac408982906300539ae6a0" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38657 } } peers { permanent_uuid: "ab8a50f16fae448da57e40b4b78bfb03" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42649 } }
I20240502 08:02:35.990461 8928 raft_consensus.cc:374] T 8ad9d4c321a449539ca014fc4e89005d P e7b4ca4d63bd42708824172a58c462a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:02:35.990473 8927 tablet_bootstrap.cc:492] T 8ad9d4c321a449539ca014fc4e89005d P ab8a50f16fae448da57e40b4b78bfb03: No bootstrap required, opened a new log
I20240502 08:02:35.991024 8928 raft_consensus.cc:729] T 8ad9d4c321a449539ca014fc4e89005d P e7b4ca4d63bd42708824172a58c462a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7b4ca4d63bd42708824172a58c462a2, State: Initialized, Role: FOLLOWER
I20240502 08:02:35.991282 8927 ts_tablet_manager.cc:1367] T 8ad9d4c321a449539ca014fc4e89005d P ab8a50f16fae448da57e40b4b78bfb03: Time spent bootstrapping tablet: real 0.024s user 0.015s sys 0.006s
I20240502 08:02:35.992274 8928 consensus_queue.cc:260] T 8ad9d4c321a449539ca014fc4e89005d P e7b4ca4d63bd42708824172a58c462a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7b4ca4d63bd42708824172a58c462a2" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 41577 } } peers { permanent_uuid: "44783b6715ac408982906300539ae6a0" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38657 } } peers { permanent_uuid: "ab8a50f16fae448da57e40b4b78bfb03" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42649 } }
I20240502 08:02:35.994103 8927 raft_consensus.cc:348] T 8ad9d4c321a449539ca014fc4e89005d P ab8a50f16fae448da57e40b4b78bfb03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7b4ca4d63bd42708824172a58c462a2" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 41577 } } peers { permanent_uuid: "44783b6715ac408982906300539ae6a0" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38657 } } peers { permanent_uuid: "ab8a50f16fae448da57e40b4b78bfb03" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42649 } }
I20240502 08:02:35.994959 8928 ts_tablet_manager.cc:1398] T 8ad9d4c321a449539ca014fc4e89005d P e7b4ca4d63bd42708824172a58c462a2: Time spent starting tablet: real 0.008s user 0.005s sys 0.001s
I20240502 08:02:35.995007 8927 raft_consensus.cc:374] T 8ad9d4c321a449539ca014fc4e89005d P ab8a50f16fae448da57e40b4b78bfb03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:02:35.995252 8929 tablet_bootstrap.cc:654] T a3b92bb4dcb7446490472014a72828dc P 44783b6715ac408982906300539ae6a0: Neither blocks nor log segments found. Creating new log.
I20240502 08:02:35.995637 8927 raft_consensus.cc:729] T 8ad9d4c321a449539ca014fc4e89005d P ab8a50f16fae448da57e40b4b78bfb03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab8a50f16fae448da57e40b4b78bfb03, State: Initialized, Role: FOLLOWER
I20240502 08:02:35.996592 8927 consensus_queue.cc:260] T 8ad9d4c321a449539ca014fc4e89005d P ab8a50f16fae448da57e40b4b78bfb03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7b4ca4d63bd42708824172a58c462a2" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 41577 } } peers { permanent_uuid: "44783b6715ac408982906300539ae6a0" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38657 } } peers { permanent_uuid: "ab8a50f16fae448da57e40b4b78bfb03" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42649 } }
I20240502 08:02:35.999193 8927 ts_tablet_manager.cc:1398] T 8ad9d4c321a449539ca014fc4e89005d P ab8a50f16fae448da57e40b4b78bfb03: Time spent starting tablet: real 0.007s user 0.001s sys 0.005s
I20240502 08:02:36.006091 8929 tablet_bootstrap.cc:492] T a3b92bb4dcb7446490472014a72828dc P 44783b6715ac408982906300539ae6a0: No bootstrap required, opened a new log
I20240502 08:02:36.006608 8929 ts_tablet_manager.cc:1367] T a3b92bb4dcb7446490472014a72828dc P 44783b6715ac408982906300539ae6a0: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.009s
I20240502 08:02:36.008725 8929 raft_consensus.cc:348] T a3b92bb4dcb7446490472014a72828dc P 44783b6715ac408982906300539ae6a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab8a50f16fae448da57e40b4b78bfb03" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42649 } } peers { permanent_uuid: "e7b4ca4d63bd42708824172a58c462a2" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 41577 } } peers { permanent_uuid: "44783b6715ac408982906300539ae6a0" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38657 } }
I20240502 08:02:36.009271 8929 raft_consensus.cc:374] T a3b92bb4dcb7446490472014a72828dc P 44783b6715ac408982906300539ae6a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:02:36.009486 8929 raft_consensus.cc:729] T a3b92bb4dcb7446490472014a72828dc P 44783b6715ac408982906300539ae6a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44783b6715ac408982906300539ae6a0, State: Initialized, Role: FOLLOWER
I20240502 08:02:36.010054 8929 consensus_queue.cc:260] T a3b92bb4dcb7446490472014a72828dc P 44783b6715ac408982906300539ae6a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab8a50f16fae448da57e40b4b78bfb03" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42649 } } peers { permanent_uuid: "e7b4ca4d63bd42708824172a58c462a2" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 41577 } } peers { permanent_uuid: "44783b6715ac408982906300539ae6a0" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38657 } }
I20240502 08:02:36.012073 8929 ts_tablet_manager.cc:1398] T a3b92bb4dcb7446490472014a72828dc P 44783b6715ac408982906300539ae6a0: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240502 08:02:36.057076 8934 raft_consensus.cc:482] T 8ad9d4c321a449539ca014fc4e89005d P ab8a50f16fae448da57e40b4b78bfb03 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:02:36.057709 8934 raft_consensus.cc:504] T 8ad9d4c321a449539ca014fc4e89005d P ab8a50f16fae448da57e40b4b78bfb03 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7b4ca4d63bd42708824172a58c462a2" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 41577 } } peers { permanent_uuid: "44783b6715ac408982906300539ae6a0" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38657 } } peers { permanent_uuid: "ab8a50f16fae448da57e40b4b78bfb03" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42649 } }
I20240502 08:02:36.060110 8934 leader_election.cc:288] T 8ad9d4c321a449539ca014fc4e89005d P ab8a50f16fae448da57e40b4b78bfb03 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7b4ca4d63bd42708824172a58c462a2 (127.6.204.194:41577), 44783b6715ac408982906300539ae6a0 (127.6.204.195:38657)
I20240502 08:02:36.071547 8882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ad9d4c321a449539ca014fc4e89005d" candidate_uuid: "ab8a50f16fae448da57e40b4b78bfb03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44783b6715ac408982906300539ae6a0" is_pre_election: true
I20240502 08:02:36.071575 8808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ad9d4c321a449539ca014fc4e89005d" candidate_uuid: "ab8a50f16fae448da57e40b4b78bfb03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7b4ca4d63bd42708824172a58c462a2" is_pre_election: true
I20240502 08:02:36.072333 8882 raft_consensus.cc:2454] T 8ad9d4c321a449539ca014fc4e89005d P 44783b6715ac408982906300539ae6a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab8a50f16fae448da57e40b4b78bfb03 in term 0.
I20240502 08:02:36.072387 8808 raft_consensus.cc:2454] T 8ad9d4c321a449539ca014fc4e89005d P e7b4ca4d63bd42708824172a58c462a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab8a50f16fae448da57e40b4b78bfb03 in term 0.
I20240502 08:02:36.073729 8696 leader_election.cc:302] T 8ad9d4c321a449539ca014fc4e89005d P ab8a50f16fae448da57e40b4b78bfb03 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab8a50f16fae448da57e40b4b78bfb03, e7b4ca4d63bd42708824172a58c462a2; no voters:
I20240502 08:02:36.074483 8934 raft_consensus.cc:2781] T 8ad9d4c321a449539ca014fc4e89005d P ab8a50f16fae448da57e40b4b78bfb03 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:02:36.074858 8934 raft_consensus.cc:482] T 8ad9d4c321a449539ca014fc4e89005d P ab8a50f16fae448da57e40b4b78bfb03 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:02:36.075132 8934 raft_consensus.cc:3037] T 8ad9d4c321a449539ca014fc4e89005d P ab8a50f16fae448da57e40b4b78bfb03 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:02:36.079699 8934 raft_consensus.cc:504] T 8ad9d4c321a449539ca014fc4e89005d P ab8a50f16fae448da57e40b4b78bfb03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7b4ca4d63bd42708824172a58c462a2" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 41577 } } peers { permanent_uuid: "44783b6715ac408982906300539ae6a0" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38657 } } peers { permanent_uuid: "ab8a50f16fae448da57e40b4b78bfb03" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42649 } }
I20240502 08:02:36.081398 8934 leader_election.cc:288] T 8ad9d4c321a449539ca014fc4e89005d P ab8a50f16fae448da57e40b4b78bfb03 [CANDIDATE]: Term 1 election: Requested vote from peers e7b4ca4d63bd42708824172a58c462a2 (127.6.204.194:41577), 44783b6715ac408982906300539ae6a0 (127.6.204.195:38657)
I20240502 08:02:36.082252 8808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ad9d4c321a449539ca014fc4e89005d" candidate_uuid: "ab8a50f16fae448da57e40b4b78bfb03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7b4ca4d63bd42708824172a58c462a2"
I20240502 08:02:36.082468 8882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ad9d4c321a449539ca014fc4e89005d" candidate_uuid: "ab8a50f16fae448da57e40b4b78bfb03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44783b6715ac408982906300539ae6a0"
I20240502 08:02:36.082902 8808 raft_consensus.cc:3037] T 8ad9d4c321a449539ca014fc4e89005d P e7b4ca4d63bd42708824172a58c462a2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:02:36.083000 8882 raft_consensus.cc:3037] T 8ad9d4c321a449539ca014fc4e89005d P 44783b6715ac408982906300539ae6a0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:02:36.089772 8882 raft_consensus.cc:2454] T 8ad9d4c321a449539ca014fc4e89005d P 44783b6715ac408982906300539ae6a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab8a50f16fae448da57e40b4b78bfb03 in term 1.
I20240502 08:02:36.089772 8808 raft_consensus.cc:2454] T 8ad9d4c321a449539ca014fc4e89005d P e7b4ca4d63bd42708824172a58c462a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab8a50f16fae448da57e40b4b78bfb03 in term 1.
I20240502 08:02:36.091071 8696 leader_election.cc:302] T 8ad9d4c321a449539ca014fc4e89005d P ab8a50f16fae448da57e40b4b78bfb03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab8a50f16fae448da57e40b4b78bfb03, e7b4ca4d63bd42708824172a58c462a2; no voters:
I20240502 08:02:36.091744 8934 raft_consensus.cc:2781] T 8ad9d4c321a449539ca014fc4e89005d P ab8a50f16fae448da57e40b4b78bfb03 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:02:36.092756 8934 raft_consensus.cc:686] T 8ad9d4c321a449539ca014fc4e89005d P ab8a50f16fae448da57e40b4b78bfb03 [term 1 LEADER]: Becoming Leader. State: Replica: ab8a50f16fae448da57e40b4b78bfb03, State: Running, Role: LEADER
I20240502 08:02:36.093401 8934 consensus_queue.cc:237] T 8ad9d4c321a449539ca014fc4e89005d P ab8a50f16fae448da57e40b4b78bfb03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7b4ca4d63bd42708824172a58c462a2" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 41577 } } peers { permanent_uuid: "44783b6715ac408982906300539ae6a0" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38657 } } peers { permanent_uuid: "ab8a50f16fae448da57e40b4b78bfb03" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42649 } }
I20240502 08:02:36.101642 8630 catalog_manager.cc:5484] T 8ad9d4c321a449539ca014fc4e89005d P ab8a50f16fae448da57e40b4b78bfb03 reported cstate change: term changed from 0 to 1, leader changed from <none> to ab8a50f16fae448da57e40b4b78bfb03 (127.6.204.193). New cstate: current_term: 1 leader_uuid: "ab8a50f16fae448da57e40b4b78bfb03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7b4ca4d63bd42708824172a58c462a2" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 41577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44783b6715ac408982906300539ae6a0" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab8a50f16fae448da57e40b4b78bfb03" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42649 } health_report { overall_health: HEALTHY } } }
I20240502 08:02:36.284143 8934 raft_consensus.cc:482] T a3b92bb4dcb7446490472014a72828dc P ab8a50f16fae448da57e40b4b78bfb03 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:02:36.284638 8934 raft_consensus.cc:504] T a3b92bb4dcb7446490472014a72828dc P ab8a50f16fae448da57e40b4b78bfb03 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab8a50f16fae448da57e40b4b78bfb03" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42649 } } peers { permanent_uuid: "e7b4ca4d63bd42708824172a58c462a2" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 41577 } } peers { permanent_uuid: "44783b6715ac408982906300539ae6a0" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38657 } }
I20240502 08:02:36.286372 8934 leader_election.cc:288] T a3b92bb4dcb7446490472014a72828dc P ab8a50f16fae448da57e40b4b78bfb03 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7b4ca4d63bd42708824172a58c462a2 (127.6.204.194:41577), 44783b6715ac408982906300539ae6a0 (127.6.204.195:38657)
I20240502 08:02:36.287247 8808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3b92bb4dcb7446490472014a72828dc" candidate_uuid: "ab8a50f16fae448da57e40b4b78bfb03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7b4ca4d63bd42708824172a58c462a2" is_pre_election: true
I20240502 08:02:36.287405 8882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3b92bb4dcb7446490472014a72828dc" candidate_uuid: "ab8a50f16fae448da57e40b4b78bfb03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44783b6715ac408982906300539ae6a0" is_pre_election: true
I20240502 08:02:36.287977 8882 raft_consensus.cc:2454] T a3b92bb4dcb7446490472014a72828dc P 44783b6715ac408982906300539ae6a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab8a50f16fae448da57e40b4b78bfb03 in term 0.
I20240502 08:02:36.287977 8808 raft_consensus.cc:2454] T a3b92bb4dcb7446490472014a72828dc P e7b4ca4d63bd42708824172a58c462a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab8a50f16fae448da57e40b4b78bfb03 in term 0.
I20240502 08:02:36.289109 8696 leader_election.cc:302] T a3b92bb4dcb7446490472014a72828dc P ab8a50f16fae448da57e40b4b78bfb03 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab8a50f16fae448da57e40b4b78bfb03, e7b4ca4d63bd42708824172a58c462a2; no voters:
I20240502 08:02:36.289780 8934 raft_consensus.cc:2781] T a3b92bb4dcb7446490472014a72828dc P ab8a50f16fae448da57e40b4b78bfb03 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:02:36.290102 8934 raft_consensus.cc:482] T a3b92bb4dcb7446490472014a72828dc P ab8a50f16fae448da57e40b4b78bfb03 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:02:36.290382 8934 raft_consensus.cc:3037] T a3b92bb4dcb7446490472014a72828dc P ab8a50f16fae448da57e40b4b78bfb03 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:02:36.294968 8934 raft_consensus.cc:504] T a3b92bb4dcb7446490472014a72828dc P ab8a50f16fae448da57e40b4b78bfb03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab8a50f16fae448da57e40b4b78bfb03" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42649 } } peers { permanent_uuid: "e7b4ca4d63bd42708824172a58c462a2" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 41577 } } peers { permanent_uuid: "44783b6715ac408982906300539ae6a0" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38657 } }
I20240502 08:02:36.296433 8934 leader_election.cc:288] T a3b92bb4dcb7446490472014a72828dc P ab8a50f16fae448da57e40b4b78bfb03 [CANDIDATE]: Term 1 election: Requested vote from peers e7b4ca4d63bd42708824172a58c462a2 (127.6.204.194:41577), 44783b6715ac408982906300539ae6a0 (127.6.204.195:38657)
I20240502 08:02:36.297333 8808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3b92bb4dcb7446490472014a72828dc" candidate_uuid: "ab8a50f16fae448da57e40b4b78bfb03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7b4ca4d63bd42708824172a58c462a2"
I20240502 08:02:36.297411 8882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3b92bb4dcb7446490472014a72828dc" candidate_uuid: "ab8a50f16fae448da57e40b4b78bfb03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44783b6715ac408982906300539ae6a0"
I20240502 08:02:36.297891 8808 raft_consensus.cc:3037] T a3b92bb4dcb7446490472014a72828dc P e7b4ca4d63bd42708824172a58c462a2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:02:36.297971 8882 raft_consensus.cc:3037] T a3b92bb4dcb7446490472014a72828dc P 44783b6715ac408982906300539ae6a0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:02:36.302178 8808 raft_consensus.cc:2454] T a3b92bb4dcb7446490472014a72828dc P e7b4ca4d63bd42708824172a58c462a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab8a50f16fae448da57e40b4b78bfb03 in term 1.
I20240502 08:02:36.302382 8882 raft_consensus.cc:2454] T a3b92bb4dcb7446490472014a72828dc P 44783b6715ac408982906300539ae6a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab8a50f16fae448da57e40b4b78bfb03 in term 1.
I20240502 08:02:36.303164 8696 leader_election.cc:302] T a3b92bb4dcb7446490472014a72828dc P ab8a50f16fae448da57e40b4b78bfb03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab8a50f16fae448da57e40b4b78bfb03, e7b4ca4d63bd42708824172a58c462a2; no voters:
I20240502 08:02:36.303867 8934 raft_consensus.cc:2781] T a3b92bb4dcb7446490472014a72828dc P ab8a50f16fae448da57e40b4b78bfb03 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:02:36.304324 8934 raft_consensus.cc:686] T a3b92bb4dcb7446490472014a72828dc P ab8a50f16fae448da57e40b4b78bfb03 [term 1 LEADER]: Becoming Leader. State: Replica: ab8a50f16fae448da57e40b4b78bfb03, State: Running, Role: LEADER
I20240502 08:02:36.304911 8934 consensus_queue.cc:237] T a3b92bb4dcb7446490472014a72828dc P ab8a50f16fae448da57e40b4b78bfb03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab8a50f16fae448da57e40b4b78bfb03" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42649 } } peers { permanent_uuid: "e7b4ca4d63bd42708824172a58c462a2" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 41577 } } peers { permanent_uuid: "44783b6715ac408982906300539ae6a0" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38657 } }
I20240502 08:02:36.311524 8630 catalog_manager.cc:5484] T a3b92bb4dcb7446490472014a72828dc P ab8a50f16fae448da57e40b4b78bfb03 reported cstate change: term changed from 0 to 1, leader changed from <none> to ab8a50f16fae448da57e40b4b78bfb03 (127.6.204.193). New cstate: current_term: 1 leader_uuid: "ab8a50f16fae448da57e40b4b78bfb03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab8a50f16fae448da57e40b4b78bfb03" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7b4ca4d63bd42708824172a58c462a2" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 41577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44783b6715ac408982906300539ae6a0" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38657 } health_report { overall_health: UNKNOWN } } }
W20240502 08:02:36.484378 8829 debug-util.cc:398] Leaking SignalData structure 0x7b080020cc60 after lost signal to thread 6966
I20240502 08:02:36.532582 8934 consensus_queue.cc:1035] T 8ad9d4c321a449539ca014fc4e89005d P ab8a50f16fae448da57e40b4b78bfb03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44783b6715ac408982906300539ae6a0" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:02:36.545027 8934 consensus_queue.cc:1035] T 8ad9d4c321a449539ca014fc4e89005d P ab8a50f16fae448da57e40b4b78bfb03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7b4ca4d63bd42708824172a58c462a2" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 41577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:02:36.591787 8684 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:02:36.606031 8733 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a3b92bb4dcb7446490472014a72828dc"
dest_uuid: "ab8a50f16fae448da57e40b4b78bfb03"
mode: GRACEFUL
new_leader_uuid: "e7b4ca4d63bd42708824172a58c462a2"
from {username='slave'} at 127.0.0.1:40894
I20240502 08:02:36.606873 8733 raft_consensus.cc:595] T a3b92bb4dcb7446490472014a72828dc P ab8a50f16fae448da57e40b4b78bfb03 [term 1 LEADER]: Received request to transfer leadership to TS e7b4ca4d63bd42708824172a58c462a2
I20240502 08:02:36.608207 8684 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240502 08:02:36.608599 8684 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:02:36.794935 8934 consensus_queue.cc:1035] T a3b92bb4dcb7446490472014a72828dc P ab8a50f16fae448da57e40b4b78bfb03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7b4ca4d63bd42708824172a58c462a2" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 41577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:02:36.801815 8941 raft_consensus.cc:979] T a3b92bb4dcb7446490472014a72828dc P ab8a50f16fae448da57e40b4b78bfb03: : Instructing follower e7b4ca4d63bd42708824172a58c462a2 to start an election
I20240502 08:02:36.802659 8934 raft_consensus.cc:1067] T a3b92bb4dcb7446490472014a72828dc P ab8a50f16fae448da57e40b4b78bfb03 [term 1 LEADER]: Signalling peer e7b4ca4d63bd42708824172a58c462a2 to start an election
I20240502 08:02:36.808810 8808 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a3b92bb4dcb7446490472014a72828dc"
dest_uuid: "e7b4ca4d63bd42708824172a58c462a2"
from {username='slave'} at 127.0.0.1:59580
I20240502 08:02:36.810171 8808 raft_consensus.cc:482] T a3b92bb4dcb7446490472014a72828dc P e7b4ca4d63bd42708824172a58c462a2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:02:36.810696 8808 raft_consensus.cc:3037] T a3b92bb4dcb7446490472014a72828dc P e7b4ca4d63bd42708824172a58c462a2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:02:36.811789 8934 consensus_queue.cc:1035] T a3b92bb4dcb7446490472014a72828dc P ab8a50f16fae448da57e40b4b78bfb03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44783b6715ac408982906300539ae6a0" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:02:36.819525 8808 raft_consensus.cc:504] T a3b92bb4dcb7446490472014a72828dc P e7b4ca4d63bd42708824172a58c462a2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab8a50f16fae448da57e40b4b78bfb03" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42649 } } peers { permanent_uuid: "e7b4ca4d63bd42708824172a58c462a2" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 41577 } } peers { permanent_uuid: "44783b6715ac408982906300539ae6a0" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38657 } }
I20240502 08:02:36.832828 8808 leader_election.cc:288] T a3b92bb4dcb7446490472014a72828dc P e7b4ca4d63bd42708824172a58c462a2 [CANDIDATE]: Term 2 election: Requested vote from peers ab8a50f16fae448da57e40b4b78bfb03 (127.6.204.193:42649), 44783b6715ac408982906300539ae6a0 (127.6.204.195:38657)
I20240502 08:02:36.838376 8733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3b92bb4dcb7446490472014a72828dc" candidate_uuid: "e7b4ca4d63bd42708824172a58c462a2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ab8a50f16fae448da57e40b4b78bfb03"
I20240502 08:02:36.839226 8733 raft_consensus.cc:3032] T a3b92bb4dcb7446490472014a72828dc P ab8a50f16fae448da57e40b4b78bfb03 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:02:36.839634 8733 raft_consensus.cc:729] T a3b92bb4dcb7446490472014a72828dc P ab8a50f16fae448da57e40b4b78bfb03 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ab8a50f16fae448da57e40b4b78bfb03, State: Running, Role: LEADER
I20240502 08:02:36.840649 8733 consensus_queue.cc:260] T a3b92bb4dcb7446490472014a72828dc P ab8a50f16fae448da57e40b4b78bfb03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab8a50f16fae448da57e40b4b78bfb03" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42649 } } peers { permanent_uuid: "e7b4ca4d63bd42708824172a58c462a2" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 41577 } } peers { permanent_uuid: "44783b6715ac408982906300539ae6a0" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38657 } }
I20240502 08:02:36.841979 8733 raft_consensus.cc:3037] T a3b92bb4dcb7446490472014a72828dc P ab8a50f16fae448da57e40b4b78bfb03 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:02:36.850436 8882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3b92bb4dcb7446490472014a72828dc" candidate_uuid: "e7b4ca4d63bd42708824172a58c462a2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "44783b6715ac408982906300539ae6a0"
I20240502 08:02:36.851003 8882 raft_consensus.cc:3037] T a3b92bb4dcb7446490472014a72828dc P 44783b6715ac408982906300539ae6a0 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:02:36.852805 8733 raft_consensus.cc:2454] T a3b92bb4dcb7446490472014a72828dc P ab8a50f16fae448da57e40b4b78bfb03 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7b4ca4d63bd42708824172a58c462a2 in term 2.
I20240502 08:02:36.854492 8774 leader_election.cc:302] T a3b92bb4dcb7446490472014a72828dc P e7b4ca4d63bd42708824172a58c462a2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab8a50f16fae448da57e40b4b78bfb03, e7b4ca4d63bd42708824172a58c462a2; no voters:
I20240502 08:02:36.855517 8948 raft_consensus.cc:2781] T a3b92bb4dcb7446490472014a72828dc P e7b4ca4d63bd42708824172a58c462a2 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:02:36.856580 8882 raft_consensus.cc:2454] T a3b92bb4dcb7446490472014a72828dc P 44783b6715ac408982906300539ae6a0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7b4ca4d63bd42708824172a58c462a2 in term 2.
I20240502 08:02:36.856957 8948 raft_consensus.cc:686] T a3b92bb4dcb7446490472014a72828dc P e7b4ca4d63bd42708824172a58c462a2 [term 2 LEADER]: Becoming Leader. State: Replica: e7b4ca4d63bd42708824172a58c462a2, State: Running, Role: LEADER
I20240502 08:02:36.857853 8948 consensus_queue.cc:237] T a3b92bb4dcb7446490472014a72828dc P e7b4ca4d63bd42708824172a58c462a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab8a50f16fae448da57e40b4b78bfb03" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42649 } } peers { permanent_uuid: "e7b4ca4d63bd42708824172a58c462a2" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 41577 } } peers { permanent_uuid: "44783b6715ac408982906300539ae6a0" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38657 } }
I20240502 08:02:36.864475 8628 catalog_manager.cc:5484] T a3b92bb4dcb7446490472014a72828dc P e7b4ca4d63bd42708824172a58c462a2 reported cstate change: term changed from 1 to 2, leader changed from ab8a50f16fae448da57e40b4b78bfb03 (127.6.204.193) to e7b4ca4d63bd42708824172a58c462a2 (127.6.204.194). New cstate: current_term: 2 leader_uuid: "e7b4ca4d63bd42708824172a58c462a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab8a50f16fae448da57e40b4b78bfb03" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7b4ca4d63bd42708824172a58c462a2" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 41577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44783b6715ac408982906300539ae6a0" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38657 } health_report { overall_health: UNKNOWN } } }
I20240502 08:02:38.161916 8733 raft_consensus.cc:1261] T a3b92bb4dcb7446490472014a72828dc P ab8a50f16fae448da57e40b4b78bfb03 [term 2 FOLLOWER]: Refusing update from remote peer e7b4ca4d63bd42708824172a58c462a2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:02:38.163926 8959 consensus_queue.cc:1035] T a3b92bb4dcb7446490472014a72828dc P e7b4ca4d63bd42708824172a58c462a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab8a50f16fae448da57e40b4b78bfb03" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240502 08:02:38.175686 8882 raft_consensus.cc:1261] T a3b92bb4dcb7446490472014a72828dc P 44783b6715ac408982906300539ae6a0 [term 2 FOLLOWER]: Refusing update from remote peer e7b4ca4d63bd42708824172a58c462a2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:02:38.177099 8959 consensus_queue.cc:1035] T a3b92bb4dcb7446490472014a72828dc P e7b4ca4d63bd42708824172a58c462a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44783b6715ac408982906300539ae6a0" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 08:02:38.235997 8754 debug-util.cc:398] Leaking SignalData structure 0x7b080011bdc0 after lost signal to thread 6966
W20240502 08:02:38.236827 8754 debug-util.cc:398] Leaking SignalData structure 0x7b08000c1560 after lost signal to thread 8665
W20240502 08:02:38.237771 8754 debug-util.cc:398] Leaking SignalData structure 0x7b08001853e0 after lost signal to thread 8757
W20240502 08:02:38.238648 8754 debug-util.cc:398] Leaking SignalData structure 0x7b08001850a0 after lost signal to thread 8832
W20240502 08:02:38.239439 8754 debug-util.cc:398] Leaking SignalData structure 0x7b080008b340 after lost signal to thread 8906
I20240502 08:02:38.609812 8684 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:02:38.612713 8684 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:02:38.613166 8684 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:02:40.614377 8684 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:02:40.617558 8684 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:02:40.618077 8684 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:02:42.619110 8684 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:02:42.621062 8684 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:02:42.621371 8684 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:02:44.622534 8684 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:02:44.624876 8684 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:02:44.625200 8684 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:02:47.039994 8684 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:02:47.042810 8684 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:02:47.043159 8684 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
W20240502 08:02:47.686371 8903 debug-util.cc:398] Leaking SignalData structure 0x7b0800053ea0 after lost signal to thread 6966
W20240502 08:02:47.687378 8903 debug-util.cc:398] Leaking SignalData structure 0x7b0800056580 after lost signal to thread 8665
W20240502 08:02:47.688320 8903 debug-util.cc:398] Leaking SignalData structure 0x7b08001fefa0 after lost signal to thread 8757
W20240502 08:02:47.689167 8903 debug-util.cc:398] Leaking SignalData structure 0x7b0800032c60 after lost signal to thread 8832
W20240502 08:02:47.690024 8903 debug-util.cc:398] Leaking SignalData structure 0x7b0800181b00 after lost signal to thread 8906
I20240502 08:02:47.821895 6963 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:02:47.830072 8978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:02:47.836215 8979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:02:47.839382 6963 server_base.cc:1034] running on GCE node
W20240502 08:02:47.839574 8981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:02:47.840821 6963 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:02:47.841070 6963 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:02:47.841269 6963 hybrid_clock.cc:648] HybridClock initialized: now 1714636967841252 us; error 0 us; skew 500 ppm
I20240502 08:02:47.841864 6963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:02:47.851238 6963 webserver.cc:460] Webserver started at http://127.6.204.196:39061/ using document root <none> and password file <none>
I20240502 08:02:47.852108 6963 fs_manager.cc:359] Metadata directory not provided
I20240502 08:02:47.852496 6963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:02:47.852882 6963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:02:47.854578 6963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714636933625983-6963-0/minicluster-data/ts-3-root/instance:
uuid: "6b3446b5c4ee412180d2d50477fe66ed"
format_stamp: "Formatted at 2024-05-02 08:02:47 on dist-test-slave-32xs"
I20240502 08:02:47.860443 6963 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240502 08:02:47.866113 8986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:02:47.867106 6963 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 08:02:47.867403 6963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714636933625983-6963-0/minicluster-data/ts-3-root
uuid: "6b3446b5c4ee412180d2d50477fe66ed"
format_stamp: "Formatted at 2024-05-02 08:02:47 on dist-test-slave-32xs"
I20240502 08:02:47.867674 6963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714636933625983-6963-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714636933625983-6963-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714636933625983-6963-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:02:47.887836 6963 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:02:47.889207 6963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:02:47.890935 6963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:02:47.893903 6963 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:02:47.894151 6963 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:02:47.894423 6963 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:02:47.894661 6963 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:02:47.952486 6963 rpc_server.cc:307] RPC server started. Bound to: 127.6.204.196:34793
I20240502 08:02:47.952594 9048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.204.196:34793 every 8 connection(s)
I20240502 08:02:47.972172 9049 heartbeater.cc:347] Connected to a master server at 127.6.204.254:36313
I20240502 08:02:47.972566 9049 heartbeater.cc:464] Registering TS with master...
I20240502 08:02:47.973330 9049 heartbeater.cc:511] Master 127.6.204.254:36313 requested a full tablet report, sending...
I20240502 08:02:47.975697 8631 ts_manager.cc:194] Registered new tserver with Master: 6b3446b5c4ee412180d2d50477fe66ed (127.6.204.196:34793)
I20240502 08:02:47.977919 8631 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52288
I20240502 08:02:48.734652 8733 consensus_queue.cc:237] T 8ad9d4c321a449539ca014fc4e89005d P ab8a50f16fae448da57e40b4b78bfb03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e7b4ca4d63bd42708824172a58c462a2" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 41577 } attrs { replace: true } } peers { permanent_uuid: "44783b6715ac408982906300539ae6a0" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38657 } } peers { permanent_uuid: "ab8a50f16fae448da57e40b4b78bfb03" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42649 } } peers { permanent_uuid: "6b3446b5c4ee412180d2d50477fe66ed" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 34793 } attrs { promote: true } }
I20240502 08:02:48.744254 8808 raft_consensus.cc:1261] T 8ad9d4c321a449539ca014fc4e89005d P e7b4ca4d63bd42708824172a58c462a2 [term 1 FOLLOWER]: Refusing update from remote peer ab8a50f16fae448da57e40b4b78bfb03: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:02:48.747448 9054 consensus_queue.cc:1035] T 8ad9d4c321a449539ca014fc4e89005d P ab8a50f16fae448da57e40b4b78bfb03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7b4ca4d63bd42708824172a58c462a2" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 41577 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240502 08:02:48.748163 8881 raft_consensus.cc:1261] T 8ad9d4c321a449539ca014fc4e89005d P 44783b6715ac408982906300539ae6a0 [term 1 FOLLOWER]: Refusing update from remote peer ab8a50f16fae448da57e40b4b78bfb03: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:02:48.750197 9054 consensus_queue.cc:1035] T 8ad9d4c321a449539ca014fc4e89005d P ab8a50f16fae448da57e40b4b78bfb03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44783b6715ac408982906300539ae6a0" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:02:48.765748 8972 raft_consensus.cc:2932] T 8ad9d4c321a449539ca014fc4e89005d P ab8a50f16fae448da57e40b4b78bfb03 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6b3446b5c4ee412180d2d50477fe66ed (127.6.204.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e7b4ca4d63bd42708824172a58c462a2" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 41577 } attrs { replace: true } } peers { permanent_uuid: "44783b6715ac408982906300539ae6a0" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38657 } } peers { permanent_uuid: "ab8a50f16fae448da57e40b4b78bfb03" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42649 } } peers { permanent_uuid: "6b3446b5c4ee412180d2d50477fe66ed" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 34793 } attrs { promote: true } } }
W20240502 08:02:48.768278 8699 consensus_peers.cc:485] T 8ad9d4c321a449539ca014fc4e89005d P ab8a50f16fae448da57e40b4b78bfb03 -> Peer 6b3446b5c4ee412180d2d50477fe66ed (127.6.204.196:34793): Couldn't send request to peer 6b3446b5c4ee412180d2d50477fe66ed. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ad9d4c321a449539ca014fc4e89005d. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:02:48.770833 8881 raft_consensus.cc:2932] T 8ad9d4c321a449539ca014fc4e89005d P 44783b6715ac408982906300539ae6a0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6b3446b5c4ee412180d2d50477fe66ed (127.6.204.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e7b4ca4d63bd42708824172a58c462a2" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 41577 } attrs { replace: true } } peers { permanent_uuid: "44783b6715ac408982906300539ae6a0" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38657 } } peers { permanent_uuid: "ab8a50f16fae448da57e40b4b78bfb03" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42649 } } peers { permanent_uuid: "6b3446b5c4ee412180d2d50477fe66ed" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 34793 } attrs { promote: true } } }
I20240502 08:02:48.773281 8808 raft_consensus.cc:2932] T 8ad9d4c321a449539ca014fc4e89005d P e7b4ca4d63bd42708824172a58c462a2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6b3446b5c4ee412180d2d50477fe66ed (127.6.204.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e7b4ca4d63bd42708824172a58c462a2" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 41577 } attrs { replace: true } } peers { permanent_uuid: "44783b6715ac408982906300539ae6a0" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38657 } } peers { permanent_uuid: "ab8a50f16fae448da57e40b4b78bfb03" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42649 } } peers { permanent_uuid: "6b3446b5c4ee412180d2d50477fe66ed" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 34793 } attrs { promote: true } } }
I20240502 08:02:48.791993 8631 catalog_manager.cc:5484] T 8ad9d4c321a449539ca014fc4e89005d P e7b4ca4d63bd42708824172a58c462a2 reported cstate change: config changed from index -1 to 2, NON_VOTER 6b3446b5c4ee412180d2d50477fe66ed (127.6.204.196) added. New cstate: current_term: 1 leader_uuid: "ab8a50f16fae448da57e40b4b78bfb03" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e7b4ca4d63bd42708824172a58c462a2" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 41577 } attrs { replace: true } } peers { permanent_uuid: "44783b6715ac408982906300539ae6a0" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38657 } } peers { permanent_uuid: "ab8a50f16fae448da57e40b4b78bfb03" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42649 } } peers { permanent_uuid: "6b3446b5c4ee412180d2d50477fe66ed" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 34793 } attrs { promote: true } } }
I20240502 08:02:48.980609 9049 heartbeater.cc:503] Master 127.6.204.254:36313 was elected leader, sending a full tablet report...
I20240502 08:02:49.044085 8684 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:02:49.046931 8684 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:02:49.047426 8684 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:02:49.219987 9068 ts_tablet_manager.cc:902] T 8ad9d4c321a449539ca014fc4e89005d P 6b3446b5c4ee412180d2d50477fe66ed: Initiating tablet copy from peer ab8a50f16fae448da57e40b4b78bfb03 (127.6.204.193:42649)
I20240502 08:02:49.225468 9068 tablet_copy_client.cc:308] T 8ad9d4c321a449539ca014fc4e89005d P 6b3446b5c4ee412180d2d50477fe66ed: tablet copy: Beginning tablet copy session from remote peer at address 127.6.204.193:42649
I20240502 08:02:49.237313 8743 tablet_copy_service.cc:140] P ab8a50f16fae448da57e40b4b78bfb03: Received BeginTabletCopySession request for tablet 8ad9d4c321a449539ca014fc4e89005d from peer 6b3446b5c4ee412180d2d50477fe66ed ({username='slave'} at 127.0.0.1:33446)
I20240502 08:02:49.237972 8743 tablet_copy_service.cc:161] P ab8a50f16fae448da57e40b4b78bfb03: Beginning new tablet copy session on tablet 8ad9d4c321a449539ca014fc4e89005d from peer 6b3446b5c4ee412180d2d50477fe66ed at {username='slave'} at 127.0.0.1:33446: session id = 6b3446b5c4ee412180d2d50477fe66ed-8ad9d4c321a449539ca014fc4e89005d
I20240502 08:02:49.245419 8743 tablet_copy_source_session.cc:205] T 8ad9d4c321a449539ca014fc4e89005d P ab8a50f16fae448da57e40b4b78bfb03: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:02:49.249231 9068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ad9d4c321a449539ca014fc4e89005d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:02:49.268019 9068 tablet_copy_client.cc:783] T 8ad9d4c321a449539ca014fc4e89005d P 6b3446b5c4ee412180d2d50477fe66ed: tablet copy: Starting download of 0 data blocks...
I20240502 08:02:49.268947 9068 tablet_copy_client.cc:647] T 8ad9d4c321a449539ca014fc4e89005d P 6b3446b5c4ee412180d2d50477fe66ed: tablet copy: Starting download of 1 WAL segments...
I20240502 08:02:49.272987 9068 tablet_copy_client.cc:523] T 8ad9d4c321a449539ca014fc4e89005d P 6b3446b5c4ee412180d2d50477fe66ed: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:02:49.283380 9068 tablet_bootstrap.cc:492] T 8ad9d4c321a449539ca014fc4e89005d P 6b3446b5c4ee412180d2d50477fe66ed: Bootstrap starting.
I20240502 08:02:49.309623 9068 tablet_bootstrap.cc:492] T 8ad9d4c321a449539ca014fc4e89005d P 6b3446b5c4ee412180d2d50477fe66ed: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:02:49.310532 9068 tablet_bootstrap.cc:492] T 8ad9d4c321a449539ca014fc4e89005d P 6b3446b5c4ee412180d2d50477fe66ed: Bootstrap complete.
I20240502 08:02:49.311271 9068 ts_tablet_manager.cc:1367] T 8ad9d4c321a449539ca014fc4e89005d P 6b3446b5c4ee412180d2d50477fe66ed: Time spent bootstrapping tablet: real 0.028s user 0.008s sys 0.016s
I20240502 08:02:49.314560 9068 raft_consensus.cc:348] T 8ad9d4c321a449539ca014fc4e89005d P 6b3446b5c4ee412180d2d50477fe66ed [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e7b4ca4d63bd42708824172a58c462a2" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 41577 } attrs { replace: true } } peers { permanent_uuid: "44783b6715ac408982906300539ae6a0" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38657 } } peers { permanent_uuid: "ab8a50f16fae448da57e40b4b78bfb03" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42649 } } peers { permanent_uuid: "6b3446b5c4ee412180d2d50477fe66ed" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 34793 } attrs { promote: true } }
I20240502 08:02:49.315387 9068 raft_consensus.cc:729] T 8ad9d4c321a449539ca014fc4e89005d P 6b3446b5c4ee412180d2d50477fe66ed [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6b3446b5c4ee412180d2d50477fe66ed, State: Initialized, Role: LEARNER
I20240502 08:02:49.316040 9068 consensus_queue.cc:260] T 8ad9d4c321a449539ca014fc4e89005d P 6b3446b5c4ee412180d2d50477fe66ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e7b4ca4d63bd42708824172a58c462a2" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 41577 } attrs { replace: true } } peers { permanent_uuid: "44783b6715ac408982906300539ae6a0" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38657 } } peers { permanent_uuid: "ab8a50f16fae448da57e40b4b78bfb03" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42649 } } peers { permanent_uuid: "6b3446b5c4ee412180d2d50477fe66ed" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 34793 } attrs { promote: true } }
I20240502 08:02:49.319046 9068 ts_tablet_manager.cc:1398] T 8ad9d4c321a449539ca014fc4e89005d P 6b3446b5c4ee412180d2d50477fe66ed: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20240502 08:02:49.323716 8743 tablet_copy_service.cc:342] P ab8a50f16fae448da57e40b4b78bfb03: Request end of tablet copy session 6b3446b5c4ee412180d2d50477fe66ed-8ad9d4c321a449539ca014fc4e89005d received from {username='slave'} at 127.0.0.1:33446
I20240502 08:02:49.324164 8743 tablet_copy_service.cc:433] P ab8a50f16fae448da57e40b4b78bfb03: ending tablet copy session 6b3446b5c4ee412180d2d50477fe66ed-8ad9d4c321a449539ca014fc4e89005d on tablet 8ad9d4c321a449539ca014fc4e89005d with peer 6b3446b5c4ee412180d2d50477fe66ed
W20240502 08:02:49.524924 8754 debug-util.cc:398] Leaking SignalData structure 0x7b080008b6e0 after lost signal to thread 6966
W20240502 08:02:49.525791 8754 debug-util.cc:398] Leaking SignalData structure 0x7b08000c45e0 after lost signal to thread 8665
W20240502 08:02:49.526901 8754 debug-util.cc:398] Leaking SignalData structure 0x7b080012e5a0 after lost signal to thread 8757
W20240502 08:02:49.527745 8754 debug-util.cc:398] Leaking SignalData structure 0x7b08000ae740 after lost signal to thread 8832
W20240502 08:02:49.528510 8754 debug-util.cc:398] Leaking SignalData structure 0x7b080020f7a0 after lost signal to thread 8906
I20240502 08:02:49.727795 9024 raft_consensus.cc:1203] T 8ad9d4c321a449539ca014fc4e89005d P 6b3446b5c4ee412180d2d50477fe66ed [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:02:50.352838 9078 raft_consensus.cc:1050] T 8ad9d4c321a449539ca014fc4e89005d P ab8a50f16fae448da57e40b4b78bfb03: attempting to promote NON_VOTER 6b3446b5c4ee412180d2d50477fe66ed to VOTER
I20240502 08:02:50.355144 9078 consensus_queue.cc:237] T 8ad9d4c321a449539ca014fc4e89005d P ab8a50f16fae448da57e40b4b78bfb03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e7b4ca4d63bd42708824172a58c462a2" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 41577 } attrs { replace: true } } peers { permanent_uuid: "44783b6715ac408982906300539ae6a0" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38657 } } peers { permanent_uuid: "ab8a50f16fae448da57e40b4b78bfb03" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42649 } } peers { permanent_uuid: "6b3446b5c4ee412180d2d50477fe66ed" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 34793 } attrs { promote: false } }
I20240502 08:02:50.361004 8881 raft_consensus.cc:1261] T 8ad9d4c321a449539ca014fc4e89005d P 44783b6715ac408982906300539ae6a0 [term 1 FOLLOWER]: Refusing update from remote peer ab8a50f16fae448da57e40b4b78bfb03: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:02:50.363489 9024 raft_consensus.cc:1261] T 8ad9d4c321a449539ca014fc4e89005d P 6b3446b5c4ee412180d2d50477fe66ed [term 1 LEARNER]: Refusing update from remote peer ab8a50f16fae448da57e40b4b78bfb03: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:02:50.365904 9081 consensus_queue.cc:1035] T 8ad9d4c321a449539ca014fc4e89005d P ab8a50f16fae448da57e40b4b78bfb03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44783b6715ac408982906300539ae6a0" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:02:50.368782 8808 raft_consensus.cc:1261] T 8ad9d4c321a449539ca014fc4e89005d P e7b4ca4d63bd42708824172a58c462a2 [term 1 FOLLOWER]: Refusing update from remote peer ab8a50f16fae448da57e40b4b78bfb03: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:02:50.372959 9078 consensus_queue.cc:1035] T 8ad9d4c321a449539ca014fc4e89005d P ab8a50f16fae448da57e40b4b78bfb03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b3446b5c4ee412180d2d50477fe66ed" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 34793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 08:02:50.377190 9081 consensus_queue.cc:1035] T 8ad9d4c321a449539ca014fc4e89005d P ab8a50f16fae448da57e40b4b78bfb03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7b4ca4d63bd42708824172a58c462a2" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 41577 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 08:02:50.394232 9078 raft_consensus.cc:2932] T 8ad9d4c321a449539ca014fc4e89005d P ab8a50f16fae448da57e40b4b78bfb03 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6b3446b5c4ee412180d2d50477fe66ed (127.6.204.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e7b4ca4d63bd42708824172a58c462a2" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 41577 } attrs { replace: true } } peers { permanent_uuid: "44783b6715ac408982906300539ae6a0" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38657 } } peers { permanent_uuid: "ab8a50f16fae448da57e40b4b78bfb03" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42649 } } peers { permanent_uuid: "6b3446b5c4ee412180d2d50477fe66ed" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 34793 } attrs { promote: false } } }
I20240502 08:02:50.401126 8808 raft_consensus.cc:2932] T 8ad9d4c321a449539ca014fc4e89005d P e7b4ca4d63bd42708824172a58c462a2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6b3446b5c4ee412180d2d50477fe66ed (127.6.204.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e7b4ca4d63bd42708824172a58c462a2" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 41577 } attrs { replace: true } } peers { permanent_uuid: "44783b6715ac408982906300539ae6a0" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38657 } } peers { permanent_uuid: "ab8a50f16fae448da57e40b4b78bfb03" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42649 } } peers { permanent_uuid: "6b3446b5c4ee412180d2d50477fe66ed" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 34793 } attrs { promote: false } } }
I20240502 08:02:50.405308 8881 raft_consensus.cc:2932] T 8ad9d4c321a449539ca014fc4e89005d P 44783b6715ac408982906300539ae6a0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6b3446b5c4ee412180d2d50477fe66ed (127.6.204.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e7b4ca4d63bd42708824172a58c462a2" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 41577 } attrs { replace: true } } peers { permanent_uuid: "44783b6715ac408982906300539ae6a0" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38657 } } peers { permanent_uuid: "ab8a50f16fae448da57e40b4b78bfb03" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42649 } } peers { permanent_uuid: "6b3446b5c4ee412180d2d50477fe66ed" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 34793 } attrs { promote: false } } }
I20240502 08:02:50.405268 9024 raft_consensus.cc:2932] T 8ad9d4c321a449539ca014fc4e89005d P 6b3446b5c4ee412180d2d50477fe66ed [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6b3446b5c4ee412180d2d50477fe66ed (127.6.204.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e7b4ca4d63bd42708824172a58c462a2" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 41577 } attrs { replace: true } } peers { permanent_uuid: "44783b6715ac408982906300539ae6a0" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38657 } } peers { permanent_uuid: "ab8a50f16fae448da57e40b4b78bfb03" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42649 } } peers { permanent_uuid: "6b3446b5c4ee412180d2d50477fe66ed" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 34793 } attrs { promote: false } } }
I20240502 08:02:50.418362 8630 catalog_manager.cc:5484] T 8ad9d4c321a449539ca014fc4e89005d P ab8a50f16fae448da57e40b4b78bfb03 reported cstate change: config changed from index 2 to 3, 6b3446b5c4ee412180d2d50477fe66ed (127.6.204.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ab8a50f16fae448da57e40b4b78bfb03" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e7b4ca4d63bd42708824172a58c462a2" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 41577 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44783b6715ac408982906300539ae6a0" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab8a50f16fae448da57e40b4b78bfb03" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b3446b5c4ee412180d2d50477fe66ed" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 34793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:02:50.451114 8733 consensus_queue.cc:237] T 8ad9d4c321a449539ca014fc4e89005d P ab8a50f16fae448da57e40b4b78bfb03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "44783b6715ac408982906300539ae6a0" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38657 } } peers { permanent_uuid: "ab8a50f16fae448da57e40b4b78bfb03" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42649 } } peers { permanent_uuid: "6b3446b5c4ee412180d2d50477fe66ed" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 34793 } attrs { promote: false } }
I20240502 08:02:50.457634 8881 raft_consensus.cc:1261] T 8ad9d4c321a449539ca014fc4e89005d P 44783b6715ac408982906300539ae6a0 [term 1 FOLLOWER]: Refusing update from remote peer ab8a50f16fae448da57e40b4b78bfb03: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:02:50.457988 9024 raft_consensus.cc:1261] T 8ad9d4c321a449539ca014fc4e89005d P 6b3446b5c4ee412180d2d50477fe66ed [term 1 FOLLOWER]: Refusing update from remote peer ab8a50f16fae448da57e40b4b78bfb03: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:02:50.459296 9081 consensus_queue.cc:1035] T 8ad9d4c321a449539ca014fc4e89005d P ab8a50f16fae448da57e40b4b78bfb03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b3446b5c4ee412180d2d50477fe66ed" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 34793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:02:50.460208 9078 consensus_queue.cc:1035] T 8ad9d4c321a449539ca014fc4e89005d P ab8a50f16fae448da57e40b4b78bfb03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44783b6715ac408982906300539ae6a0" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:02:50.468086 9088 raft_consensus.cc:2932] T 8ad9d4c321a449539ca014fc4e89005d P ab8a50f16fae448da57e40b4b78bfb03 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e7b4ca4d63bd42708824172a58c462a2 (127.6.204.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "44783b6715ac408982906300539ae6a0" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38657 } } peers { permanent_uuid: "ab8a50f16fae448da57e40b4b78bfb03" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42649 } } peers { permanent_uuid: "6b3446b5c4ee412180d2d50477fe66ed" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 34793 } attrs { promote: false } } }
I20240502 08:02:50.470525 8881 raft_consensus.cc:2932] T 8ad9d4c321a449539ca014fc4e89005d P 44783b6715ac408982906300539ae6a0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e7b4ca4d63bd42708824172a58c462a2 (127.6.204.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "44783b6715ac408982906300539ae6a0" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38657 } } peers { permanent_uuid: "ab8a50f16fae448da57e40b4b78bfb03" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42649 } } peers { permanent_uuid: "6b3446b5c4ee412180d2d50477fe66ed" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 34793 } attrs { promote: false } } }
I20240502 08:02:50.472924 9024 raft_consensus.cc:2932] T 8ad9d4c321a449539ca014fc4e89005d P 6b3446b5c4ee412180d2d50477fe66ed [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e7b4ca4d63bd42708824172a58c462a2 (127.6.204.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "44783b6715ac408982906300539ae6a0" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38657 } } peers { permanent_uuid: "ab8a50f16fae448da57e40b4b78bfb03" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42649 } } peers { permanent_uuid: "6b3446b5c4ee412180d2d50477fe66ed" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 34793 } attrs { promote: false } } }
I20240502 08:02:50.478554 8618 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8ad9d4c321a449539ca014fc4e89005d with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:02:50.483933 8628 catalog_manager.cc:5484] T 8ad9d4c321a449539ca014fc4e89005d P 44783b6715ac408982906300539ae6a0 reported cstate change: config changed from index 3 to 4, VOTER e7b4ca4d63bd42708824172a58c462a2 (127.6.204.194) evicted. New cstate: current_term: 1 leader_uuid: "ab8a50f16fae448da57e40b4b78bfb03" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "44783b6715ac408982906300539ae6a0" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38657 } } peers { permanent_uuid: "ab8a50f16fae448da57e40b4b78bfb03" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 42649 } } peers { permanent_uuid: "6b3446b5c4ee412180d2d50477fe66ed" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 34793 } attrs { promote: false } } }
I20240502 08:02:50.497273 8798 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ad9d4c321a449539ca014fc4e89005d with delete_type TABLET_DATA_TOMBSTONED (TS e7b4ca4d63bd42708824172a58c462a2 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:59568
I20240502 08:02:50.505338 9095 tablet_replica.cc:330] T 8ad9d4c321a449539ca014fc4e89005d P e7b4ca4d63bd42708824172a58c462a2: stopping tablet replica
I20240502 08:02:50.506356 9095 raft_consensus.cc:2229] T 8ad9d4c321a449539ca014fc4e89005d P e7b4ca4d63bd42708824172a58c462a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:02:50.507138 9095 raft_consensus.cc:2258] T 8ad9d4c321a449539ca014fc4e89005d P e7b4ca4d63bd42708824172a58c462a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:02:50.510393 9095 ts_tablet_manager.cc:1875] T 8ad9d4c321a449539ca014fc4e89005d P e7b4ca4d63bd42708824172a58c462a2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:02:50.526997 9095 ts_tablet_manager.cc:1888] T 8ad9d4c321a449539ca014fc4e89005d P e7b4ca4d63bd42708824172a58c462a2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 08:02:50.527457 9095 log.cc:1196] T 8ad9d4c321a449539ca014fc4e89005d P e7b4ca4d63bd42708824172a58c462a2: Deleting WAL directory at /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714636933625983-6963-0/minicluster-data/ts-1-root/wals/8ad9d4c321a449539ca014fc4e89005d
I20240502 08:02:50.529178 8615 catalog_manager.cc:4830] TS e7b4ca4d63bd42708824172a58c462a2 (127.6.204.194:41577): tablet 8ad9d4c321a449539ca014fc4e89005d (table test-workload [id=6f7ba7649f0648ff9867b9a0d015ff38]) successfully deleted
I20240502 08:02:51.048451 8684 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:02:51.050628 8684 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:02:51.050935 8684 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:02:53.051755 8684 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:02:53.053560 8684 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:02:53.053838 8684 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:02:55.054924 8684 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:02:55.057143 8684 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:02:55.057451 8684 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:02:57.058348 8684 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:02:57.060261 8684 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:02:57.060544 8684 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:02:59.061447 8684 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:02:59.063359 8684 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:02:59.063622 8684 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:03:01.064553 8684 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:03:01.066498 8684 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:03:01.066781 8684 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:03:03.067713 8684 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:03:03.069809 8684 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:03:03.070158 8684 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:03:03.338598 6963 tablet_server.cc:178] TabletServer@127.6.204.193:0 shutting down...
I20240502 08:03:03.368546 6963 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 08:03:03.369290 6963 tablet_replica.cc:330] T 8ad9d4c321a449539ca014fc4e89005d P ab8a50f16fae448da57e40b4b78bfb03: stopping tablet replica
I20240502 08:03:03.369995 6963 raft_consensus.cc:2229] T 8ad9d4c321a449539ca014fc4e89005d P ab8a50f16fae448da57e40b4b78bfb03 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:03:03.371037 6963 raft_consensus.cc:2258] T 8ad9d4c321a449539ca014fc4e89005d P ab8a50f16fae448da57e40b4b78bfb03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:03:03.373111 6963 tablet_replica.cc:330] T a3b92bb4dcb7446490472014a72828dc P ab8a50f16fae448da57e40b4b78bfb03: stopping tablet replica
I20240502 08:03:03.373558 6963 raft_consensus.cc:2229] T a3b92bb4dcb7446490472014a72828dc P ab8a50f16fae448da57e40b4b78bfb03 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:03:03.373983 6963 raft_consensus.cc:2258] T a3b92bb4dcb7446490472014a72828dc P ab8a50f16fae448da57e40b4b78bfb03 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:03:03.421582 6963 tablet_server.cc:194] TabletServer@127.6.204.193:0 shutdown complete.
I20240502 08:03:03.435894 6963 tablet_server.cc:178] TabletServer@127.6.204.194:0 shutting down...
I20240502 08:03:03.462036 6963 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 08:03:03.462872 6963 tablet_replica.cc:330] T a3b92bb4dcb7446490472014a72828dc P e7b4ca4d63bd42708824172a58c462a2: stopping tablet replica
I20240502 08:03:03.463354 6963 raft_consensus.cc:2229] T a3b92bb4dcb7446490472014a72828dc P e7b4ca4d63bd42708824172a58c462a2 [term 2 LEADER]: Raft consensus shutting down.
I20240502 08:03:03.464228 6963 raft_consensus.cc:2258] T a3b92bb4dcb7446490472014a72828dc P e7b4ca4d63bd42708824172a58c462a2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:03:03.502502 6963 tablet_server.cc:194] TabletServer@127.6.204.194:0 shutdown complete.
I20240502 08:03:03.514272 6963 tablet_server.cc:178] TabletServer@127.6.204.195:0 shutting down...
I20240502 08:03:03.541966 6963 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 08:03:03.542588 6963 tablet_replica.cc:330] T a3b92bb4dcb7446490472014a72828dc P 44783b6715ac408982906300539ae6a0: stopping tablet replica
I20240502 08:03:03.543107 6963 raft_consensus.cc:2229] T a3b92bb4dcb7446490472014a72828dc P 44783b6715ac408982906300539ae6a0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:03:03.543572 6963 raft_consensus.cc:2258] T a3b92bb4dcb7446490472014a72828dc P 44783b6715ac408982906300539ae6a0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:03:03.545858 6963 tablet_replica.cc:330] T 8ad9d4c321a449539ca014fc4e89005d P 44783b6715ac408982906300539ae6a0: stopping tablet replica
I20240502 08:03:03.546375 6963 raft_consensus.cc:2229] T 8ad9d4c321a449539ca014fc4e89005d P 44783b6715ac408982906300539ae6a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:03:03.546936 6963 raft_consensus.cc:2258] T 8ad9d4c321a449539ca014fc4e89005d P 44783b6715ac408982906300539ae6a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:03:03.583873 6963 tablet_server.cc:194] TabletServer@127.6.204.195:0 shutdown complete.
I20240502 08:03:03.596467 6963 tablet_server.cc:178] TabletServer@127.6.204.196:0 shutting down...
I20240502 08:03:03.617642 6963 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 08:03:03.618263 6963 tablet_replica.cc:330] T 8ad9d4c321a449539ca014fc4e89005d P 6b3446b5c4ee412180d2d50477fe66ed: stopping tablet replica
I20240502 08:03:03.618765 6963 raft_consensus.cc:2229] T 8ad9d4c321a449539ca014fc4e89005d P 6b3446b5c4ee412180d2d50477fe66ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:03:03.619318 6963 raft_consensus.cc:2258] T 8ad9d4c321a449539ca014fc4e89005d P 6b3446b5c4ee412180d2d50477fe66ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:03:03.637938 6963 tablet_server.cc:194] TabletServer@127.6.204.196:0 shutdown complete.
I20240502 08:03:03.649680 6963 master.cc:537] Master@127.6.204.254:36313 shutting down...
I20240502 08:03:03.667954 6963 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 4479d3bf0da04a05a10cf6d93d86ef40 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:03:03.668540 6963 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 4479d3bf0da04a05a10cf6d93d86ef40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:03:03.668865 6963 tablet_replica.cc:330] T 00000000000000000000000000000000 P 4479d3bf0da04a05a10cf6d93d86ef40: stopping tablet replica
I20240502 08:03:03.687067 6963 master.cc:552] Master@127.6.204.254:36313 shutdown complete.
[ OK ] AutoRebalancerTest.MovesScheduledIfAddTserver (29261 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfNoTservers
I20240502 08:03:03.724396 6963 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.6.204.254:39093
I20240502 08:03:03.725493 6963 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:03:03.730954 9102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:03:03.731292 9103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:03:03.733456 9105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:03:03.734454 6963 server_base.cc:1034] running on GCE node
I20240502 08:03:03.735495 6963 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:03:03.735688 6963 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:03:03.735858 6963 hybrid_clock.cc:648] HybridClock initialized: now 1714636983735843 us; error 0 us; skew 500 ppm
I20240502 08:03:03.736377 6963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:03:03.738981 6963 webserver.cc:460] Webserver started at http://127.6.204.254:41323/ using document root <none> and password file <none>
I20240502 08:03:03.739451 6963 fs_manager.cc:359] Metadata directory not provided
I20240502 08:03:03.739619 6963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:03:03.739887 6963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:03:03.741119 6963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714636933625983-6963-0/minicluster-data/master-0-root/instance:
uuid: "d84b20a3f314491c8ce036fbc6878d2c"
format_stamp: "Formatted at 2024-05-02 08:03:03 on dist-test-slave-32xs"
I20240502 08:03:03.745934 6963 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240502 08:03:03.749353 9110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:03:03.750038 6963 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 08:03:03.750370 6963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714636933625983-6963-0/minicluster-data/master-0-root
uuid: "d84b20a3f314491c8ce036fbc6878d2c"
format_stamp: "Formatted at 2024-05-02 08:03:03 on dist-test-slave-32xs"
I20240502 08:03:03.750640 6963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714636933625983-6963-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714636933625983-6963-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714636933625983-6963-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:03:03.759922 6963 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:03:03.761088 6963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:03:03.797904 6963 rpc_server.cc:307] RPC server started. Bound to: 127.6.204.254:39093
I20240502 08:03:03.797995 9161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.204.254:39093 every 8 connection(s)
I20240502 08:03:03.801807 9162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:03:03.812137 9162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d84b20a3f314491c8ce036fbc6878d2c: Bootstrap starting.
I20240502 08:03:03.816402 9162 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d84b20a3f314491c8ce036fbc6878d2c: Neither blocks nor log segments found. Creating new log.
I20240502 08:03:03.820658 9162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d84b20a3f314491c8ce036fbc6878d2c: No bootstrap required, opened a new log
I20240502 08:03:03.822748 9162 raft_consensus.cc:348] T 00000000000000000000000000000000 P d84b20a3f314491c8ce036fbc6878d2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d84b20a3f314491c8ce036fbc6878d2c" member_type: VOTER }
I20240502 08:03:03.823210 9162 raft_consensus.cc:374] T 00000000000000000000000000000000 P d84b20a3f314491c8ce036fbc6878d2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:03:03.823395 9162 raft_consensus.cc:729] T 00000000000000000000000000000000 P d84b20a3f314491c8ce036fbc6878d2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d84b20a3f314491c8ce036fbc6878d2c, State: Initialized, Role: FOLLOWER
I20240502 08:03:03.823866 9162 consensus_queue.cc:260] T 00000000000000000000000000000000 P d84b20a3f314491c8ce036fbc6878d2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d84b20a3f314491c8ce036fbc6878d2c" member_type: VOTER }
I20240502 08:03:03.824293 9162 raft_consensus.cc:388] T 00000000000000000000000000000000 P d84b20a3f314491c8ce036fbc6878d2c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:03:03.824482 9162 raft_consensus.cc:482] T 00000000000000000000000000000000 P d84b20a3f314491c8ce036fbc6878d2c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:03:03.824704 9162 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d84b20a3f314491c8ce036fbc6878d2c [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:03:03.828966 9162 raft_consensus.cc:504] T 00000000000000000000000000000000 P d84b20a3f314491c8ce036fbc6878d2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d84b20a3f314491c8ce036fbc6878d2c" member_type: VOTER }
I20240502 08:03:03.829438 9162 leader_election.cc:302] T 00000000000000000000000000000000 P d84b20a3f314491c8ce036fbc6878d2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d84b20a3f314491c8ce036fbc6878d2c; no voters:
I20240502 08:03:03.830569 9162 leader_election.cc:288] T 00000000000000000000000000000000 P d84b20a3f314491c8ce036fbc6878d2c [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:03:03.830960 9165 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d84b20a3f314491c8ce036fbc6878d2c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:03:03.832310 9165 raft_consensus.cc:686] T 00000000000000000000000000000000 P d84b20a3f314491c8ce036fbc6878d2c [term 1 LEADER]: Becoming Leader. State: Replica: d84b20a3f314491c8ce036fbc6878d2c, State: Running, Role: LEADER
I20240502 08:03:03.833097 9165 consensus_queue.cc:237] T 00000000000000000000000000000000 P d84b20a3f314491c8ce036fbc6878d2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d84b20a3f314491c8ce036fbc6878d2c" member_type: VOTER }
I20240502 08:03:03.833786 9162 sys_catalog.cc:564] T 00000000000000000000000000000000 P d84b20a3f314491c8ce036fbc6878d2c [sys.catalog]: configured and running, proceeding with master startup.
I20240502 08:03:03.838326 9166 sys_catalog.cc:455] T 00000000000000000000000000000000 P d84b20a3f314491c8ce036fbc6878d2c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d84b20a3f314491c8ce036fbc6878d2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d84b20a3f314491c8ce036fbc6878d2c" member_type: VOTER } }
I20240502 08:03:03.838624 9167 sys_catalog.cc:455] T 00000000000000000000000000000000 P d84b20a3f314491c8ce036fbc6878d2c [sys.catalog]: SysCatalogTable state changed. Reason: New leader d84b20a3f314491c8ce036fbc6878d2c. Latest consensus state: current_term: 1 leader_uuid: "d84b20a3f314491c8ce036fbc6878d2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d84b20a3f314491c8ce036fbc6878d2c" member_type: VOTER } }
I20240502 08:03:03.839283 9166 sys_catalog.cc:458] T 00000000000000000000000000000000 P d84b20a3f314491c8ce036fbc6878d2c [sys.catalog]: This master's current role is: LEADER
I20240502 08:03:03.839391 9167 sys_catalog.cc:458] T 00000000000000000000000000000000 P d84b20a3f314491c8ce036fbc6878d2c [sys.catalog]: This master's current role is: LEADER
I20240502 08:03:03.842757 9173 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 08:03:03.847568 9173 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 08:03:03.854249 6963 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 08:03:03.857812 9173 catalog_manager.cc:1347] Generated new cluster ID: 22ba6292a5524584bac64a53aed00a26
I20240502 08:03:03.858083 9173 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 08:03:03.884564 9173 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 08:03:03.885861 9173 catalog_manager.cc:1504] Loading token signing keys...
I20240502 08:03:03.899850 9173 catalog_manager.cc:5857] T 00000000000000000000000000000000 P d84b20a3f314491c8ce036fbc6878d2c: Generated new TSK 0
I20240502 08:03:03.900442 9173 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 08:03:03.923477 6963 internal_mini_cluster.cc:371] 0 TS(s) registered with all masters after 0.000214509s
I20240502 08:03:16.052273 6963 master.cc:537] Master@127.6.204.254:39093 shutting down...
I20240502 08:03:16.077694 6963 raft_consensus.cc:2229] T 00000000000000000000000000000000 P d84b20a3f314491c8ce036fbc6878d2c [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:03:16.078598 6963 raft_consensus.cc:2258] T 00000000000000000000000000000000 P d84b20a3f314491c8ce036fbc6878d2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:03:16.078949 6963 tablet_replica.cc:330] T 00000000000000000000000000000000 P d84b20a3f314491c8ce036fbc6878d2c: stopping tablet replica
W20240502 08:03:17.011916 9158 debug-util.cc:398] Leaking SignalData structure 0x7b080019d2c0 after lost signal to thread 6966
W20240502 08:03:17.097373 6963 thread.cc:536] Waited for 1000ms trying to join with diag-logger (tid 9158)
W20240502 08:03:18.097880 6963 thread.cc:536] Waited for 2000ms trying to join with diag-logger (tid 9158)
W20240502 08:03:19.098364 6963 thread.cc:536] Waited for 3000ms trying to join with diag-logger (tid 9158)
I20240502 08:03:19.210968 6963 master.cc:552] Master@127.6.204.254:39093 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfNoTservers (15516 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfNoTablets
I20240502 08:03:19.241093 6963 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.6.204.254:35795
I20240502 08:03:19.242187 6963 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:03:19.247691 9186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:03:19.248404 9187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:03:19.249298 9189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:03:19.250280 6963 server_base.cc:1034] running on GCE node
I20240502 08:03:19.252161 6963 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:03:19.252347 6963 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:03:19.252462 6963 hybrid_clock.cc:648] HybridClock initialized: now 1714636999252452 us; error 0 us; skew 500 ppm
I20240502 08:03:19.252895 6963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:03:19.259568 6963 webserver.cc:460] Webserver started at http://127.6.204.254:46405/ using document root <none> and password file <none>
I20240502 08:03:19.260063 6963 fs_manager.cc:359] Metadata directory not provided
I20240502 08:03:19.260224 6963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:03:19.260484 6963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:03:19.261817 6963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714636933625983-6963-0/minicluster-data/master-0-root/instance:
uuid: "dd91bd75c4e24256b93d8198b1e286fd"
format_stamp: "Formatted at 2024-05-02 08:03:19 on dist-test-slave-32xs"
I20240502 08:03:19.266539 6963 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240502 08:03:19.269940 9194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:03:19.270728 6963 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 08:03:19.270996 6963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714636933625983-6963-0/minicluster-data/master-0-root
uuid: "dd91bd75c4e24256b93d8198b1e286fd"
format_stamp: "Formatted at 2024-05-02 08:03:19 on dist-test-slave-32xs"
I20240502 08:03:19.271308 6963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714636933625983-6963-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714636933625983-6963-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714636933625983-6963-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:03:19.286829 6963 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:03:19.288031 6963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:03:19.325922 6963 rpc_server.cc:307] RPC server started. Bound to: 127.6.204.254:35795
I20240502 08:03:19.326006 9245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.204.254:35795 every 8 connection(s)
I20240502 08:03:19.329741 9246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:03:19.340839 9246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd91bd75c4e24256b93d8198b1e286fd: Bootstrap starting.
I20240502 08:03:19.345181 9246 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd91bd75c4e24256b93d8198b1e286fd: Neither blocks nor log segments found. Creating new log.
I20240502 08:03:19.349339 9246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd91bd75c4e24256b93d8198b1e286fd: No bootstrap required, opened a new log
I20240502 08:03:19.351336 9246 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd91bd75c4e24256b93d8198b1e286fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd91bd75c4e24256b93d8198b1e286fd" member_type: VOTER }
I20240502 08:03:19.351828 9246 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd91bd75c4e24256b93d8198b1e286fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:03:19.352036 9246 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd91bd75c4e24256b93d8198b1e286fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd91bd75c4e24256b93d8198b1e286fd, State: Initialized, Role: FOLLOWER
I20240502 08:03:19.352563 9246 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd91bd75c4e24256b93d8198b1e286fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd91bd75c4e24256b93d8198b1e286fd" member_type: VOTER }
I20240502 08:03:19.353005 9246 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd91bd75c4e24256b93d8198b1e286fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:03:19.353215 9246 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd91bd75c4e24256b93d8198b1e286fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:03:19.353471 9246 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd91bd75c4e24256b93d8198b1e286fd [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:03:19.358501 9246 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd91bd75c4e24256b93d8198b1e286fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd91bd75c4e24256b93d8198b1e286fd" member_type: VOTER }
I20240502 08:03:19.359030 9246 leader_election.cc:302] T 00000000000000000000000000000000 P dd91bd75c4e24256b93d8198b1e286fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd91bd75c4e24256b93d8198b1e286fd; no voters:
I20240502 08:03:19.360159 9246 leader_election.cc:288] T 00000000000000000000000000000000 P dd91bd75c4e24256b93d8198b1e286fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:03:19.360565 9249 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd91bd75c4e24256b93d8198b1e286fd [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:03:19.361827 9249 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd91bd75c4e24256b93d8198b1e286fd [term 1 LEADER]: Becoming Leader. State: Replica: dd91bd75c4e24256b93d8198b1e286fd, State: Running, Role: LEADER
I20240502 08:03:19.362550 9249 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd91bd75c4e24256b93d8198b1e286fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd91bd75c4e24256b93d8198b1e286fd" member_type: VOTER }
I20240502 08:03:19.363281 9246 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd91bd75c4e24256b93d8198b1e286fd [sys.catalog]: configured and running, proceeding with master startup.
I20240502 08:03:19.368058 9251 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd91bd75c4e24256b93d8198b1e286fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd91bd75c4e24256b93d8198b1e286fd. Latest consensus state: current_term: 1 leader_uuid: "dd91bd75c4e24256b93d8198b1e286fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd91bd75c4e24256b93d8198b1e286fd" member_type: VOTER } }
I20240502 08:03:19.368402 9250 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd91bd75c4e24256b93d8198b1e286fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd91bd75c4e24256b93d8198b1e286fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd91bd75c4e24256b93d8198b1e286fd" member_type: VOTER } }
I20240502 08:03:19.368911 9250 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd91bd75c4e24256b93d8198b1e286fd [sys.catalog]: This master's current role is: LEADER
I20240502 08:03:19.368893 9251 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd91bd75c4e24256b93d8198b1e286fd [sys.catalog]: This master's current role is: LEADER
I20240502 08:03:19.371181 9256 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 08:03:19.379562 9256 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 08:03:19.380574 6963 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 08:03:19.388386 9256 catalog_manager.cc:1347] Generated new cluster ID: 54d5348b2281436981f3cd722c383dc9
I20240502 08:03:19.388670 9256 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 08:03:19.409607 9256 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 08:03:19.410955 9256 catalog_manager.cc:1504] Loading token signing keys...
I20240502 08:03:19.428032 9256 catalog_manager.cc:5857] T 00000000000000000000000000000000 P dd91bd75c4e24256b93d8198b1e286fd: Generated new TSK 0
I20240502 08:03:19.428700 9256 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 08:03:19.446952 6963 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:03:19.453109 9267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:03:19.454033 9268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:03:19.455495 9270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:03:19.456301 6963 server_base.cc:1034] running on GCE node
I20240502 08:03:19.457154 6963 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:03:19.457336 6963 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:03:19.457458 6963 hybrid_clock.cc:648] HybridClock initialized: now 1714636999457447 us; error 0 us; skew 500 ppm
I20240502 08:03:19.457926 6963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:03:19.460387 6963 webserver.cc:460] Webserver started at http://127.6.204.193:43265/ using document root <none> and password file <none>
I20240502 08:03:19.460850 6963 fs_manager.cc:359] Metadata directory not provided
I20240502 08:03:19.461014 6963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:03:19.461231 6963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:03:19.462430 6963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714636933625983-6963-0/minicluster-data/ts-0-root/instance:
uuid: "fcb3fb5e27124739abccb509126c279c"
format_stamp: "Formatted at 2024-05-02 08:03:19 on dist-test-slave-32xs"
I20240502 08:03:19.466949 6963 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20240502 08:03:19.470233 9275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:03:19.471000 6963 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20240502 08:03:19.471258 6963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714636933625983-6963-0/minicluster-data/ts-0-root
uuid: "fcb3fb5e27124739abccb509126c279c"
format_stamp: "Formatted at 2024-05-02 08:03:19 on dist-test-slave-32xs"
I20240502 08:03:19.471504 6963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714636933625983-6963-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714636933625983-6963-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714636933625983-6963-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:03:19.521469 6963 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:03:19.522729 6963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:03:19.524247 6963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:03:19.526628 6963 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:03:19.526818 6963 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:03:19.527041 6963 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:03:19.527211 6963 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:03:19.568001 6963 rpc_server.cc:307] RPC server started. Bound to: 127.6.204.193:44131
I20240502 08:03:19.568105 9337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.204.193:44131 every 8 connection(s)
I20240502 08:03:19.573069 6963 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:03:19.580477 9342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:03:19.582458 9343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:03:19.586880 9345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:03:19.587749 6963 server_base.cc:1034] running on GCE node
I20240502 08:03:19.588733 6963 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:03:19.588927 6963 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:03:19.589113 6963 hybrid_clock.cc:648] HybridClock initialized: now 1714636999589093 us; error 0 us; skew 500 ppm
I20240502 08:03:19.589382 9338 heartbeater.cc:347] Connected to a master server at 127.6.204.254:35795
I20240502 08:03:19.589699 6963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:03:19.589758 9338 heartbeater.cc:464] Registering TS with master...
I20240502 08:03:19.590688 9338 heartbeater.cc:511] Master 127.6.204.254:35795 requested a full tablet report, sending...
I20240502 08:03:19.592545 6963 webserver.cc:460] Webserver started at http://127.6.204.194:38243/ using document root <none> and password file <none>
I20240502 08:03:19.593194 6963 fs_manager.cc:359] Metadata directory not provided
I20240502 08:03:19.593199 9211 ts_manager.cc:194] Registered new tserver with Master: fcb3fb5e27124739abccb509126c279c (127.6.204.193:44131)
I20240502 08:03:19.593569 6963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:03:19.593931 6963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:03:19.595682 6963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714636933625983-6963-0/minicluster-data/ts-1-root/instance:
uuid: "4329992920df435190243b76638d111e"
format_stamp: "Formatted at 2024-05-02 08:03:19 on dist-test-slave-32xs"
I20240502 08:03:19.595906 9211 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37996
I20240502 08:03:19.600630 6963 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20240502 08:03:19.603962 9350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:03:19.604707 6963 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 08:03:19.604964 6963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714636933625983-6963-0/minicluster-data/ts-1-root
uuid: "4329992920df435190243b76638d111e"
format_stamp: "Formatted at 2024-05-02 08:03:19 on dist-test-slave-32xs"
I20240502 08:03:19.605230 6963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714636933625983-6963-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714636933625983-6963-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714636933625983-6963-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:03:19.623858 6963 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:03:19.625057 6963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:03:19.626483 6963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:03:19.628671 6963 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:03:19.628886 6963 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:03:19.629097 6963 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:03:19.629249 6963 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:03:19.669090 6963 rpc_server.cc:307] RPC server started. Bound to: 127.6.204.194:35041
I20240502 08:03:19.669201 9412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.204.194:35041 every 8 connection(s)
I20240502 08:03:19.674185 6963 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:03:19.680887 9416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:03:19.683354 9417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:03:19.685267 9419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:03:19.685874 6963 server_base.cc:1034] running on GCE node
I20240502 08:03:19.686735 6963 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:03:19.687007 6963 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:03:19.687218 6963 hybrid_clock.cc:648] HybridClock initialized: now 1714636999687201 us; error 0 us; skew 500 ppm
I20240502 08:03:19.687234 9413 heartbeater.cc:347] Connected to a master server at 127.6.204.254:35795
I20240502 08:03:19.687681 9413 heartbeater.cc:464] Registering TS with master...
I20240502 08:03:19.687873 6963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:03:19.688477 9413 heartbeater.cc:511] Master 127.6.204.254:35795 requested a full tablet report, sending...
I20240502 08:03:19.690658 9211 ts_manager.cc:194] Registered new tserver with Master: 4329992920df435190243b76638d111e (127.6.204.194:35041)
I20240502 08:03:19.690876 6963 webserver.cc:460] Webserver started at http://127.6.204.195:42287/ using document root <none> and password file <none>
I20240502 08:03:19.691499 6963 fs_manager.cc:359] Metadata directory not provided
I20240502 08:03:19.691730 6963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:03:19.692073 6963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:03:19.692548 9211 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34200
I20240502 08:03:19.693653 6963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714636933625983-6963-0/minicluster-data/ts-2-root/instance:
uuid: "0dfb82d4364546c89794d584c73baa19"
format_stamp: "Formatted at 2024-05-02 08:03:19 on dist-test-slave-32xs"
I20240502 08:03:19.698402 6963 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20240502 08:03:19.701848 9424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:03:19.702623 6963 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 08:03:19.702903 6963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714636933625983-6963-0/minicluster-data/ts-2-root
uuid: "0dfb82d4364546c89794d584c73baa19"
format_stamp: "Formatted at 2024-05-02 08:03:19 on dist-test-slave-32xs"
I20240502 08:03:19.703173 6963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714636933625983-6963-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714636933625983-6963-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714636933625983-6963-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:03:19.725201 6963 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:03:19.726452 6963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:03:19.727880 6963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:03:19.730190 6963 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:03:19.730383 6963 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:03:19.730599 6963 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:03:19.730751 6963 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:03:19.770788 6963 rpc_server.cc:307] RPC server started. Bound to: 127.6.204.195:39049
I20240502 08:03:19.770887 9486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.204.195:39049 every 8 connection(s)
I20240502 08:03:19.784814 9487 heartbeater.cc:347] Connected to a master server at 127.6.204.254:35795
I20240502 08:03:19.785195 9487 heartbeater.cc:464] Registering TS with master...
I20240502 08:03:19.785940 9487 heartbeater.cc:511] Master 127.6.204.254:35795 requested a full tablet report, sending...
I20240502 08:03:19.787726 9211 ts_manager.cc:194] Registered new tserver with Master: 0dfb82d4364546c89794d584c73baa19 (127.6.204.195:39049)
I20240502 08:03:19.788501 6963 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.014414925s
I20240502 08:03:19.789283 9211 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34216
I20240502 08:03:20.598860 9338 heartbeater.cc:503] Master 127.6.204.254:35795 was elected leader, sending a full tablet report...
I20240502 08:03:20.695106 9413 heartbeater.cc:503] Master 127.6.204.254:35795 was elected leader, sending a full tablet report...
I20240502 08:03:20.792415 9487 heartbeater.cc:503] Master 127.6.204.254:35795 was elected leader, sending a full tablet report...
I20240502 08:03:21.379778 9264 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:03:23.380630 9264 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:03:25.381695 9264 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:03:27.382889 9264 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
W20240502 08:03:27.670030 9483 debug-util.cc:398] Leaking SignalData structure 0x7b080016a280 after lost signal to thread 6966
W20240502 08:03:27.671003 9483 debug-util.cc:398] Leaking SignalData structure 0x7b0800167a20 after lost signal to thread 9245
W20240502 08:03:27.671967 9483 debug-util.cc:398] Leaking SignalData structure 0x7b080008a160 after lost signal to thread 9337
W20240502 08:03:27.673008 9483 debug-util.cc:398] Leaking SignalData structure 0x7b0800058d80 after lost signal to thread 9412
W20240502 08:03:27.673949 9483 debug-util.cc:398] Leaking SignalData structure 0x7b08001936c0 after lost signal to thread 9486
I20240502 08:03:29.383800 9264 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:03:31.384756 9264 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:03:32.033914 6963 tablet_server.cc:178] TabletServer@127.6.204.193:0 shutting down...
I20240502 08:03:32.059614 6963 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 08:03:32.076699 6963 tablet_server.cc:194] TabletServer@127.6.204.193:0 shutdown complete.
I20240502 08:03:32.086284 6963 tablet_server.cc:178] TabletServer@127.6.204.194:0 shutting down...
I20240502 08:03:32.105681 6963 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 08:03:32.121407 6963 tablet_server.cc:194] TabletServer@127.6.204.194:0 shutdown complete.
I20240502 08:03:32.130208 6963 tablet_server.cc:178] TabletServer@127.6.204.195:0 shutting down...
I20240502 08:03:32.148741 6963 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 08:03:32.181820 6963 tablet_server.cc:194] TabletServer@127.6.204.195:0 shutdown complete.
I20240502 08:03:32.190330 6963 master.cc:537] Master@127.6.204.254:35795 shutting down...
I20240502 08:03:32.207547 6963 raft_consensus.cc:2229] T 00000000000000000000000000000000 P dd91bd75c4e24256b93d8198b1e286fd [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:03:32.208293 6963 raft_consensus.cc:2258] T 00000000000000000000000000000000 P dd91bd75c4e24256b93d8198b1e286fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:03:32.208588 6963 tablet_replica.cc:330] T 00000000000000000000000000000000 P dd91bd75c4e24256b93d8198b1e286fd: stopping tablet replica
I20240502 08:03:32.226563 6963 master.cc:552] Master@127.6.204.254:35795 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfNoTablets (13014 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne
I20240502 08:03:32.255635 6963 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.6.204.254:43021
I20240502 08:03:32.256687 6963 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:03:32.262480 9493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:03:32.262640 9492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:03:32.263684 9495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:03:32.264823 6963 server_base.cc:1034] running on GCE node
I20240502 08:03:32.265583 6963 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:03:32.265779 6963 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:03:32.265906 6963 hybrid_clock.cc:648] HybridClock initialized: now 1714637012265890 us; error 0 us; skew 500 ppm
I20240502 08:03:32.266484 6963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:03:32.274212 6963 webserver.cc:460] Webserver started at http://127.6.204.254:33707/ using document root <none> and password file <none>
I20240502 08:03:32.274683 6963 fs_manager.cc:359] Metadata directory not provided
I20240502 08:03:32.274852 6963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:03:32.275112 6963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:03:32.276382 6963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714636933625983-6963-0/minicluster-data/master-0-root/instance:
uuid: "36815569922d42b594948340ac5b11df"
format_stamp: "Formatted at 2024-05-02 08:03:32 on dist-test-slave-32xs"
I20240502 08:03:32.281348 6963 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20240502 08:03:32.284817 9500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:03:32.285594 6963 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 08:03:32.285841 6963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714636933625983-6963-0/minicluster-data/master-0-root
uuid: "36815569922d42b594948340ac5b11df"
format_stamp: "Formatted at 2024-05-02 08:03:32 on dist-test-slave-32xs"
I20240502 08:03:32.286109 6963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714636933625983-6963-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714636933625983-6963-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714636933625983-6963-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:03:32.297611 6963 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:03:32.298780 6963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:03:32.335868 6963 rpc_server.cc:307] RPC server started. Bound to: 127.6.204.254:43021
I20240502 08:03:32.336035 9551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.204.254:43021 every 8 connection(s)
I20240502 08:03:32.339742 9552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:03:32.354233 9552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36815569922d42b594948340ac5b11df: Bootstrap starting.
I20240502 08:03:32.358533 9552 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36815569922d42b594948340ac5b11df: Neither blocks nor log segments found. Creating new log.
I20240502 08:03:32.362705 9552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36815569922d42b594948340ac5b11df: No bootstrap required, opened a new log
I20240502 08:03:32.364703 9552 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36815569922d42b594948340ac5b11df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36815569922d42b594948340ac5b11df" member_type: VOTER }
I20240502 08:03:32.365206 9552 raft_consensus.cc:374] T 00000000000000000000000000000000 P 36815569922d42b594948340ac5b11df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:03:32.365386 9552 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36815569922d42b594948340ac5b11df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36815569922d42b594948340ac5b11df, State: Initialized, Role: FOLLOWER
I20240502 08:03:32.365906 9552 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36815569922d42b594948340ac5b11df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36815569922d42b594948340ac5b11df" member_type: VOTER }
I20240502 08:03:32.366423 9552 raft_consensus.cc:388] T 00000000000000000000000000000000 P 36815569922d42b594948340ac5b11df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:03:32.366617 9552 raft_consensus.cc:482] T 00000000000000000000000000000000 P 36815569922d42b594948340ac5b11df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:03:32.366837 9552 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36815569922d42b594948340ac5b11df [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:03:32.371316 9552 raft_consensus.cc:504] T 00000000000000000000000000000000 P 36815569922d42b594948340ac5b11df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36815569922d42b594948340ac5b11df" member_type: VOTER }
I20240502 08:03:32.371758 9552 leader_election.cc:302] T 00000000000000000000000000000000 P 36815569922d42b594948340ac5b11df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36815569922d42b594948340ac5b11df; no voters:
I20240502 08:03:32.372807 9552 leader_election.cc:288] T 00000000000000000000000000000000 P 36815569922d42b594948340ac5b11df [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:03:32.373198 9555 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 36815569922d42b594948340ac5b11df [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:03:32.374595 9555 raft_consensus.cc:686] T 00000000000000000000000000000000 P 36815569922d42b594948340ac5b11df [term 1 LEADER]: Becoming Leader. State: Replica: 36815569922d42b594948340ac5b11df, State: Running, Role: LEADER
I20240502 08:03:32.375275 9555 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36815569922d42b594948340ac5b11df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36815569922d42b594948340ac5b11df" member_type: VOTER }
I20240502 08:03:32.375799 9552 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36815569922d42b594948340ac5b11df [sys.catalog]: configured and running, proceeding with master startup.
I20240502 08:03:32.381057 9557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36815569922d42b594948340ac5b11df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36815569922d42b594948340ac5b11df. Latest consensus state: current_term: 1 leader_uuid: "36815569922d42b594948340ac5b11df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36815569922d42b594948340ac5b11df" member_type: VOTER } }
I20240502 08:03:32.381250 9556 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36815569922d42b594948340ac5b11df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "36815569922d42b594948340ac5b11df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36815569922d42b594948340ac5b11df" member_type: VOTER } }
I20240502 08:03:32.381937 9557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36815569922d42b594948340ac5b11df [sys.catalog]: This master's current role is: LEADER
I20240502 08:03:32.381999 9556 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36815569922d42b594948340ac5b11df [sys.catalog]: This master's current role is: LEADER
I20240502 08:03:32.384145 9564 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 08:03:32.389139 9564 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 08:03:32.391110 6963 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 08:03:32.397605 9564 catalog_manager.cc:1347] Generated new cluster ID: a73e515e05784221854f1ba4f9078957
I20240502 08:03:32.397936 9564 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 08:03:32.446260 9564 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 08:03:32.447561 9564 catalog_manager.cc:1504] Loading token signing keys...
I20240502 08:03:32.460393 9564 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 36815569922d42b594948340ac5b11df: Generated new TSK 0
I20240502 08:03:32.460999 9564 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 08:03:32.521975 6963 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:03:32.527683 9573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:03:32.528611 9574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:03:32.530165 9576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:03:32.531322 6963 server_base.cc:1034] running on GCE node
I20240502 08:03:32.532462 6963 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:03:32.532655 6963 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:03:32.532809 6963 hybrid_clock.cc:648] HybridClock initialized: now 1714637012532790 us; error 0 us; skew 500 ppm
I20240502 08:03:32.533310 6963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:03:32.536038 6963 webserver.cc:460] Webserver started at http://127.6.204.193:36285/ using document root <none> and password file <none>
I20240502 08:03:32.536598 6963 fs_manager.cc:359] Metadata directory not provided
I20240502 08:03:32.536790 6963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:03:32.537071 6963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:03:32.538374 6963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714636933625983-6963-0/minicluster-data/ts-0-root/instance:
uuid: "62f75069b443426690d0fbc60773d0b6"
format_stamp: "Formatted at 2024-05-02 08:03:32 on dist-test-slave-32xs"
I20240502 08:03:32.542887 6963 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240502 08:03:32.546178 9581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:03:32.546883 6963 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 08:03:32.547158 6963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714636933625983-6963-0/minicluster-data/ts-0-root
uuid: "62f75069b443426690d0fbc60773d0b6"
format_stamp: "Formatted at 2024-05-02 08:03:32 on dist-test-slave-32xs"
I20240502 08:03:32.547430 6963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714636933625983-6963-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714636933625983-6963-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714636933625983-6963-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:03:32.558418 6963 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:03:32.559576 6963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:03:32.561127 6963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:03:32.563516 6963 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:03:32.563714 6963 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:03:32.563931 6963 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:03:32.564100 6963 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:03:32.604269 6963 rpc_server.cc:307] RPC server started. Bound to: 127.6.204.193:33309
I20240502 08:03:32.604367 9643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.204.193:33309 every 8 connection(s)
I20240502 08:03:32.609978 6963 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:03:32.616607 9648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:03:32.617281 9649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:03:32.620198 9651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:03:32.620447 6963 server_base.cc:1034] running on GCE node
I20240502 08:03:32.621337 6963 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:03:32.621626 6963 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:03:32.621834 6963 hybrid_clock.cc:648] HybridClock initialized: now 1714637012621815 us; error 0 us; skew 500 ppm
I20240502 08:03:32.622445 6963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:03:32.622854 9644 heartbeater.cc:347] Connected to a master server at 127.6.204.254:43021
I20240502 08:03:32.623222 9644 heartbeater.cc:464] Registering TS with master...
I20240502 08:03:32.624203 9644 heartbeater.cc:511] Master 127.6.204.254:43021 requested a full tablet report, sending...
I20240502 08:03:32.625231 6963 webserver.cc:460] Webserver started at http://127.6.204.194:35945/ using document root <none> and password file <none>
I20240502 08:03:32.625792 6963 fs_manager.cc:359] Metadata directory not provided
I20240502 08:03:32.626011 6963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:03:32.626287 6963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:03:32.626406 9517 ts_manager.cc:194] Registered new tserver with Master: 62f75069b443426690d0fbc60773d0b6 (127.6.204.193:33309)
I20240502 08:03:32.627473 6963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714636933625983-6963-0/minicluster-data/ts-1-root/instance:
uuid: "4d753a6ad0e5457d91fadd5c694cdd57"
format_stamp: "Formatted at 2024-05-02 08:03:32 on dist-test-slave-32xs"
I20240502 08:03:32.628262 9517 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56052
I20240502 08:03:32.632411 6963 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240502 08:03:32.635818 9656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:03:32.636624 6963 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 08:03:32.636898 6963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714636933625983-6963-0/minicluster-data/ts-1-root
uuid: "4d753a6ad0e5457d91fadd5c694cdd57"
format_stamp: "Formatted at 2024-05-02 08:03:32 on dist-test-slave-32xs"
I20240502 08:03:32.637183 6963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714636933625983-6963-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714636933625983-6963-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714636933625983-6963-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:03:32.656625 6963 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:03:32.657845 6963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:03:32.659232 6963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:03:32.661445 6963 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:03:32.661636 6963 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:03:32.661855 6963 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:03:32.662004 6963 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:03:32.700836 6963 rpc_server.cc:307] RPC server started. Bound to: 127.6.204.194:39169
I20240502 08:03:32.700950 9718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.204.194:39169 every 8 connection(s)
I20240502 08:03:32.705422 6963 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:03:32.711896 9722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:03:32.713622 9723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:03:32.715692 6963 server_base.cc:1034] running on GCE node
W20240502 08:03:32.715751 9725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:03:32.716931 6963 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:03:32.717198 6963 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:03:32.717411 6963 hybrid_clock.cc:648] HybridClock initialized: now 1714637012717393 us; error 0 us; skew 500 ppm
I20240502 08:03:32.718079 6963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:03:32.718072 9719 heartbeater.cc:347] Connected to a master server at 127.6.204.254:43021
I20240502 08:03:32.718603 9719 heartbeater.cc:464] Registering TS with master...
I20240502 08:03:32.719410 9719 heartbeater.cc:511] Master 127.6.204.254:43021 requested a full tablet report, sending...
I20240502 08:03:32.721266 6963 webserver.cc:460] Webserver started at http://127.6.204.195:45041/ using document root <none> and password file <none>
I20240502 08:03:32.721498 9517 ts_manager.cc:194] Registered new tserver with Master: 4d753a6ad0e5457d91fadd5c694cdd57 (127.6.204.194:39169)
I20240502 08:03:32.721750 6963 fs_manager.cc:359] Metadata directory not provided
I20240502 08:03:32.722004 6963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:03:32.722280 6963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:03:32.722985 9517 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56060
I20240502 08:03:32.723477 6963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714636933625983-6963-0/minicluster-data/ts-2-root/instance:
uuid: "2452c4a20183446fa5aca63766a39cf8"
format_stamp: "Formatted at 2024-05-02 08:03:32 on dist-test-slave-32xs"
I20240502 08:03:32.728003 6963 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 08:03:32.731261 9730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:03:32.732035 6963 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 08:03:32.732311 6963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714636933625983-6963-0/minicluster-data/ts-2-root
uuid: "2452c4a20183446fa5aca63766a39cf8"
format_stamp: "Formatted at 2024-05-02 08:03:32 on dist-test-slave-32xs"
I20240502 08:03:32.732573 6963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714636933625983-6963-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714636933625983-6963-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714636933625983-6963-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:03:32.746479 6963 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:03:32.747673 6963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:03:32.749155 6963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:03:32.751422 6963 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:03:32.751612 6963 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:03:32.751830 6963 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:03:32.751982 6963 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:03:32.790860 6963 rpc_server.cc:307] RPC server started. Bound to: 127.6.204.195:37683
I20240502 08:03:32.790958 9792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.204.195:37683 every 8 connection(s)
I20240502 08:03:32.795428 6963 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:03:32.801889 9796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:03:32.804529 9797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:03:32.807299 9793 heartbeater.cc:347] Connected to a master server at 127.6.204.254:43021
I20240502 08:03:32.807487 6963 server_base.cc:1034] running on GCE node
I20240502 08:03:32.807765 9793 heartbeater.cc:464] Registering TS with master...
W20240502 08:03:32.807653 9799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:03:32.808590 9793 heartbeater.cc:511] Master 127.6.204.254:43021 requested a full tablet report, sending...
I20240502 08:03:32.808708 6963 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:03:32.808919 6963 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:03:32.809046 6963 hybrid_clock.cc:648] HybridClock initialized: now 1714637012809035 us; error 0 us; skew 500 ppm
I20240502 08:03:32.809667 6963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:03:32.810487 9517 ts_manager.cc:194] Registered new tserver with Master: 2452c4a20183446fa5aca63766a39cf8 (127.6.204.195:37683)
I20240502 08:03:32.811961 9517 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56076
I20240502 08:03:32.812393 6963 webserver.cc:460] Webserver started at http://127.6.204.196:33977/ using document root <none> and password file <none>
I20240502 08:03:32.812963 6963 fs_manager.cc:359] Metadata directory not provided
I20240502 08:03:32.813184 6963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:03:32.813434 6963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:03:32.814541 6963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714636933625983-6963-0/minicluster-data/ts-3-root/instance:
uuid: "763aeeacf0c74a82a9fdc368b40e9cd4"
format_stamp: "Formatted at 2024-05-02 08:03:32 on dist-test-slave-32xs"
I20240502 08:03:32.818773 6963 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20240502 08:03:32.821842 9804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:03:32.822646 6963 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 08:03:32.822935 6963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714636933625983-6963-0/minicluster-data/ts-3-root
uuid: "763aeeacf0c74a82a9fdc368b40e9cd4"
format_stamp: "Formatted at 2024-05-02 08:03:32 on dist-test-slave-32xs"
I20240502 08:03:32.823218 6963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714636933625983-6963-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714636933625983-6963-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714636933625983-6963-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:03:32.846032 6963 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:03:32.847266 6963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:03:32.848776 6963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:03:32.851038 6963 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:03:32.851243 6963 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:03:32.851464 6963 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:03:32.851614 6963 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:03:32.892932 6963 rpc_server.cc:307] RPC server started. Bound to: 127.6.204.196:42581
I20240502 08:03:32.893029 9866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.204.196:42581 every 8 connection(s)
I20240502 08:03:32.906195 9867 heartbeater.cc:347] Connected to a master server at 127.6.204.254:43021
I20240502 08:03:32.906579 9867 heartbeater.cc:464] Registering TS with master...
I20240502 08:03:32.907286 9867 heartbeater.cc:511] Master 127.6.204.254:43021 requested a full tablet report, sending...
I20240502 08:03:32.909054 9517 ts_manager.cc:194] Registered new tserver with Master: 763aeeacf0c74a82a9fdc368b40e9cd4 (127.6.204.196:42581)
I20240502 08:03:32.910020 6963 internal_mini_cluster.cc:371] 4 TS(s) registered with all masters after 0.013857089s
I20240502 08:03:32.910447 9517 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56084
I20240502 08:03:33.442276 6963 test_util.cc:268] Using random seed: 169533135
I20240502 08:03:33.463918 9517 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56104:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:03:33.509759 9832 tablet_service.cc:1467] Processing CreateTablet for tablet f85e880e71c54e72a5218f2f058e1848 (DEFAULT_TABLE table=test-workload [id=46b3bc6ef14a4fdab4dd7bb9a680f898]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:03:33.510973 9832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f85e880e71c54e72a5218f2f058e1848. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:03:33.516930 9684 tablet_service.cc:1467] Processing CreateTablet for tablet f85e880e71c54e72a5218f2f058e1848 (DEFAULT_TABLE table=test-workload [id=46b3bc6ef14a4fdab4dd7bb9a680f898]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:03:33.516930 9609 tablet_service.cc:1467] Processing CreateTablet for tablet f85e880e71c54e72a5218f2f058e1848 (DEFAULT_TABLE table=test-workload [id=46b3bc6ef14a4fdab4dd7bb9a680f898]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:03:33.518175 9684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f85e880e71c54e72a5218f2f058e1848. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:03:33.518452 9609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f85e880e71c54e72a5218f2f058e1848. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:03:33.528429 9888 tablet_bootstrap.cc:492] T f85e880e71c54e72a5218f2f058e1848 P 763aeeacf0c74a82a9fdc368b40e9cd4: Bootstrap starting.
I20240502 08:03:33.536775 9888 tablet_bootstrap.cc:654] T f85e880e71c54e72a5218f2f058e1848 P 763aeeacf0c74a82a9fdc368b40e9cd4: Neither blocks nor log segments found. Creating new log.
I20240502 08:03:33.537583 9890 tablet_bootstrap.cc:492] T f85e880e71c54e72a5218f2f058e1848 P 62f75069b443426690d0fbc60773d0b6: Bootstrap starting.
I20240502 08:03:33.538821 9889 tablet_bootstrap.cc:492] T f85e880e71c54e72a5218f2f058e1848 P 4d753a6ad0e5457d91fadd5c694cdd57: Bootstrap starting.
I20240502 08:03:33.543005 9888 tablet_bootstrap.cc:492] T f85e880e71c54e72a5218f2f058e1848 P 763aeeacf0c74a82a9fdc368b40e9cd4: No bootstrap required, opened a new log
I20240502 08:03:33.543416 9888 ts_tablet_manager.cc:1367] T f85e880e71c54e72a5218f2f058e1848 P 763aeeacf0c74a82a9fdc368b40e9cd4: Time spent bootstrapping tablet: real 0.015s user 0.012s sys 0.000s
I20240502 08:03:33.544307 9890 tablet_bootstrap.cc:654] T f85e880e71c54e72a5218f2f058e1848 P 62f75069b443426690d0fbc60773d0b6: Neither blocks nor log segments found. Creating new log.
I20240502 08:03:33.545311 9889 tablet_bootstrap.cc:654] T f85e880e71c54e72a5218f2f058e1848 P 4d753a6ad0e5457d91fadd5c694cdd57: Neither blocks nor log segments found. Creating new log.
I20240502 08:03:33.545890 9888 raft_consensus.cc:348] T f85e880e71c54e72a5218f2f058e1848 P 763aeeacf0c74a82a9fdc368b40e9cd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763aeeacf0c74a82a9fdc368b40e9cd4" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 42581 } } peers { permanent_uuid: "4d753a6ad0e5457d91fadd5c694cdd57" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39169 } } peers { permanent_uuid: "62f75069b443426690d0fbc60773d0b6" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 33309 } }
I20240502 08:03:33.546782 9888 raft_consensus.cc:374] T f85e880e71c54e72a5218f2f058e1848 P 763aeeacf0c74a82a9fdc368b40e9cd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:03:33.547143 9888 raft_consensus.cc:729] T f85e880e71c54e72a5218f2f058e1848 P 763aeeacf0c74a82a9fdc368b40e9cd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 763aeeacf0c74a82a9fdc368b40e9cd4, State: Initialized, Role: FOLLOWER
I20240502 08:03:33.547967 9888 consensus_queue.cc:260] T f85e880e71c54e72a5218f2f058e1848 P 763aeeacf0c74a82a9fdc368b40e9cd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763aeeacf0c74a82a9fdc368b40e9cd4" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 42581 } } peers { permanent_uuid: "4d753a6ad0e5457d91fadd5c694cdd57" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39169 } } peers { permanent_uuid: "62f75069b443426690d0fbc60773d0b6" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 33309 } }
I20240502 08:03:33.551434 9867 heartbeater.cc:503] Master 127.6.204.254:43021 was elected leader, sending a full tablet report...
I20240502 08:03:33.551868 9888 ts_tablet_manager.cc:1398] T f85e880e71c54e72a5218f2f058e1848 P 763aeeacf0c74a82a9fdc368b40e9cd4: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20240502 08:03:33.553668 9890 tablet_bootstrap.cc:492] T f85e880e71c54e72a5218f2f058e1848 P 62f75069b443426690d0fbc60773d0b6: No bootstrap required, opened a new log
I20240502 08:03:33.554060 9890 ts_tablet_manager.cc:1367] T f85e880e71c54e72a5218f2f058e1848 P 62f75069b443426690d0fbc60773d0b6: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.007s
I20240502 08:03:33.554656 9889 tablet_bootstrap.cc:492] T f85e880e71c54e72a5218f2f058e1848 P 4d753a6ad0e5457d91fadd5c694cdd57: No bootstrap required, opened a new log
I20240502 08:03:33.555167 9889 ts_tablet_manager.cc:1367] T f85e880e71c54e72a5218f2f058e1848 P 4d753a6ad0e5457d91fadd5c694cdd57: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.003s
I20240502 08:03:33.556301 9890 raft_consensus.cc:348] T f85e880e71c54e72a5218f2f058e1848 P 62f75069b443426690d0fbc60773d0b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763aeeacf0c74a82a9fdc368b40e9cd4" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 42581 } } peers { permanent_uuid: "4d753a6ad0e5457d91fadd5c694cdd57" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39169 } } peers { permanent_uuid: "62f75069b443426690d0fbc60773d0b6" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 33309 } }
I20240502 08:03:33.556836 9890 raft_consensus.cc:374] T f85e880e71c54e72a5218f2f058e1848 P 62f75069b443426690d0fbc60773d0b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:03:33.557067 9890 raft_consensus.cc:729] T f85e880e71c54e72a5218f2f058e1848 P 62f75069b443426690d0fbc60773d0b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62f75069b443426690d0fbc60773d0b6, State: Initialized, Role: FOLLOWER
I20240502 08:03:33.557664 9890 consensus_queue.cc:260] T f85e880e71c54e72a5218f2f058e1848 P 62f75069b443426690d0fbc60773d0b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763aeeacf0c74a82a9fdc368b40e9cd4" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 42581 } } peers { permanent_uuid: "4d753a6ad0e5457d91fadd5c694cdd57" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39169 } } peers { permanent_uuid: "62f75069b443426690d0fbc60773d0b6" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 33309 } }
I20240502 08:03:33.558746 9889 raft_consensus.cc:348] T f85e880e71c54e72a5218f2f058e1848 P 4d753a6ad0e5457d91fadd5c694cdd57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763aeeacf0c74a82a9fdc368b40e9cd4" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 42581 } } peers { permanent_uuid: "4d753a6ad0e5457d91fadd5c694cdd57" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39169 } } peers { permanent_uuid: "62f75069b443426690d0fbc60773d0b6" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 33309 } }
I20240502 08:03:33.559634 9889 raft_consensus.cc:374] T f85e880e71c54e72a5218f2f058e1848 P 4d753a6ad0e5457d91fadd5c694cdd57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:03:33.559985 9889 raft_consensus.cc:729] T f85e880e71c54e72a5218f2f058e1848 P 4d753a6ad0e5457d91fadd5c694cdd57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d753a6ad0e5457d91fadd5c694cdd57, State: Initialized, Role: FOLLOWER
I20240502 08:03:33.560937 9890 ts_tablet_manager.cc:1398] T f85e880e71c54e72a5218f2f058e1848 P 62f75069b443426690d0fbc60773d0b6: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240502 08:03:33.560732 9889 consensus_queue.cc:260] T f85e880e71c54e72a5218f2f058e1848 P 4d753a6ad0e5457d91fadd5c694cdd57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763aeeacf0c74a82a9fdc368b40e9cd4" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 42581 } } peers { permanent_uuid: "4d753a6ad0e5457d91fadd5c694cdd57" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39169 } } peers { permanent_uuid: "62f75069b443426690d0fbc60773d0b6" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 33309 } }
I20240502 08:03:33.561574 9644 heartbeater.cc:503] Master 127.6.204.254:43021 was elected leader, sending a full tablet report...
I20240502 08:03:33.563709 9889 ts_tablet_manager.cc:1398] T f85e880e71c54e72a5218f2f058e1848 P 4d753a6ad0e5457d91fadd5c694cdd57: Time spent starting tablet: real 0.008s user 0.005s sys 0.001s
I20240502 08:03:33.563997 9719 heartbeater.cc:503] Master 127.6.204.254:43021 was elected leader, sending a full tablet report...
W20240502 08:03:33.611833 9645 tablet.cc:2369] T f85e880e71c54e72a5218f2f058e1848 P 62f75069b443426690d0fbc60773d0b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 08:03:33.800222 9894 raft_consensus.cc:482] T f85e880e71c54e72a5218f2f058e1848 P 763aeeacf0c74a82a9fdc368b40e9cd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:03:33.800757 9894 raft_consensus.cc:504] T f85e880e71c54e72a5218f2f058e1848 P 763aeeacf0c74a82a9fdc368b40e9cd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763aeeacf0c74a82a9fdc368b40e9cd4" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 42581 } } peers { permanent_uuid: "4d753a6ad0e5457d91fadd5c694cdd57" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39169 } } peers { permanent_uuid: "62f75069b443426690d0fbc60773d0b6" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 33309 } }
I20240502 08:03:33.802735 9894 leader_election.cc:288] T f85e880e71c54e72a5218f2f058e1848 P 763aeeacf0c74a82a9fdc368b40e9cd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d753a6ad0e5457d91fadd5c694cdd57 (127.6.204.194:39169), 62f75069b443426690d0fbc60773d0b6 (127.6.204.193:33309)
I20240502 08:03:33.813764 9619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85e880e71c54e72a5218f2f058e1848" candidate_uuid: "763aeeacf0c74a82a9fdc368b40e9cd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62f75069b443426690d0fbc60773d0b6" is_pre_election: true
I20240502 08:03:33.814397 9793 heartbeater.cc:503] Master 127.6.204.254:43021 was elected leader, sending a full tablet report...
I20240502 08:03:33.814155 9694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85e880e71c54e72a5218f2f058e1848" candidate_uuid: "763aeeacf0c74a82a9fdc368b40e9cd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d753a6ad0e5457d91fadd5c694cdd57" is_pre_election: true
I20240502 08:03:33.814754 9619 raft_consensus.cc:2454] T f85e880e71c54e72a5218f2f058e1848 P 62f75069b443426690d0fbc60773d0b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 763aeeacf0c74a82a9fdc368b40e9cd4 in term 0.
I20240502 08:03:33.815122 9694 raft_consensus.cc:2454] T f85e880e71c54e72a5218f2f058e1848 P 4d753a6ad0e5457d91fadd5c694cdd57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 763aeeacf0c74a82a9fdc368b40e9cd4 in term 0.
I20240502 08:03:33.816154 9807 leader_election.cc:302] T f85e880e71c54e72a5218f2f058e1848 P 763aeeacf0c74a82a9fdc368b40e9cd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62f75069b443426690d0fbc60773d0b6, 763aeeacf0c74a82a9fdc368b40e9cd4; no voters:
I20240502 08:03:33.816972 9894 raft_consensus.cc:2781] T f85e880e71c54e72a5218f2f058e1848 P 763aeeacf0c74a82a9fdc368b40e9cd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:03:33.817360 9894 raft_consensus.cc:482] T f85e880e71c54e72a5218f2f058e1848 P 763aeeacf0c74a82a9fdc368b40e9cd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:03:33.817704 9894 raft_consensus.cc:3037] T f85e880e71c54e72a5218f2f058e1848 P 763aeeacf0c74a82a9fdc368b40e9cd4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:03:33.822750 9894 raft_consensus.cc:504] T f85e880e71c54e72a5218f2f058e1848 P 763aeeacf0c74a82a9fdc368b40e9cd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763aeeacf0c74a82a9fdc368b40e9cd4" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 42581 } } peers { permanent_uuid: "4d753a6ad0e5457d91fadd5c694cdd57" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39169 } } peers { permanent_uuid: "62f75069b443426690d0fbc60773d0b6" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 33309 } }
I20240502 08:03:33.824362 9894 leader_election.cc:288] T f85e880e71c54e72a5218f2f058e1848 P 763aeeacf0c74a82a9fdc368b40e9cd4 [CANDIDATE]: Term 1 election: Requested vote from peers 4d753a6ad0e5457d91fadd5c694cdd57 (127.6.204.194:39169), 62f75069b443426690d0fbc60773d0b6 (127.6.204.193:33309)
I20240502 08:03:33.825141 9694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85e880e71c54e72a5218f2f058e1848" candidate_uuid: "763aeeacf0c74a82a9fdc368b40e9cd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d753a6ad0e5457d91fadd5c694cdd57"
I20240502 08:03:33.825376 9619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85e880e71c54e72a5218f2f058e1848" candidate_uuid: "763aeeacf0c74a82a9fdc368b40e9cd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62f75069b443426690d0fbc60773d0b6"
I20240502 08:03:33.825637 9694 raft_consensus.cc:3037] T f85e880e71c54e72a5218f2f058e1848 P 4d753a6ad0e5457d91fadd5c694cdd57 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:03:33.825997 9619 raft_consensus.cc:3037] T f85e880e71c54e72a5218f2f058e1848 P 62f75069b443426690d0fbc60773d0b6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:03:33.830631 9694 raft_consensus.cc:2454] T f85e880e71c54e72a5218f2f058e1848 P 4d753a6ad0e5457d91fadd5c694cdd57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 763aeeacf0c74a82a9fdc368b40e9cd4 in term 1.
I20240502 08:03:33.830961 9619 raft_consensus.cc:2454] T f85e880e71c54e72a5218f2f058e1848 P 62f75069b443426690d0fbc60773d0b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 763aeeacf0c74a82a9fdc368b40e9cd4 in term 1.
I20240502 08:03:33.831701 9806 leader_election.cc:302] T f85e880e71c54e72a5218f2f058e1848 P 763aeeacf0c74a82a9fdc368b40e9cd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d753a6ad0e5457d91fadd5c694cdd57, 763aeeacf0c74a82a9fdc368b40e9cd4; no voters:
I20240502 08:03:33.832481 9894 raft_consensus.cc:2781] T f85e880e71c54e72a5218f2f058e1848 P 763aeeacf0c74a82a9fdc368b40e9cd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:03:33.833739 9894 raft_consensus.cc:686] T f85e880e71c54e72a5218f2f058e1848 P 763aeeacf0c74a82a9fdc368b40e9cd4 [term 1 LEADER]: Becoming Leader. State: Replica: 763aeeacf0c74a82a9fdc368b40e9cd4, State: Running, Role: LEADER
I20240502 08:03:33.834478 9894 consensus_queue.cc:237] T f85e880e71c54e72a5218f2f058e1848 P 763aeeacf0c74a82a9fdc368b40e9cd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763aeeacf0c74a82a9fdc368b40e9cd4" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 42581 } } peers { permanent_uuid: "4d753a6ad0e5457d91fadd5c694cdd57" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39169 } } peers { permanent_uuid: "62f75069b443426690d0fbc60773d0b6" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 33309 } }
I20240502 08:03:33.841794 9516 catalog_manager.cc:5484] T f85e880e71c54e72a5218f2f058e1848 P 763aeeacf0c74a82a9fdc368b40e9cd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 763aeeacf0c74a82a9fdc368b40e9cd4 (127.6.204.196). New cstate: current_term: 1 leader_uuid: "763aeeacf0c74a82a9fdc368b40e9cd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763aeeacf0c74a82a9fdc368b40e9cd4" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 42581 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d753a6ad0e5457d91fadd5c694cdd57" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "62f75069b443426690d0fbc60773d0b6" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 33309 } health_report { overall_health: UNKNOWN } } }
I20240502 08:03:34.254925 9894 consensus_queue.cc:1035] T f85e880e71c54e72a5218f2f058e1848 P 763aeeacf0c74a82a9fdc368b40e9cd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d753a6ad0e5457d91fadd5c694cdd57" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 08:03:34.273129 9894 consensus_queue.cc:1035] T f85e880e71c54e72a5218f2f058e1848 P 763aeeacf0c74a82a9fdc368b40e9cd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62f75069b443426690d0fbc60773d0b6" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 08:03:34.390956 9570 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:03:34.392347 9570 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:03:34.392669 9570 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:03:36.393592 9570 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:03:36.394937 9570 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:03:36.395236 9570 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:03:38.396131 9570 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:03:38.397455 9570 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:03:38.397756 9570 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:03:40.398761 9570 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:03:40.400121 9570 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:03:40.400466 9570 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:03:42.401373 9570 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:03:42.402706 9570 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:03:42.403168 9570 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:03:44.404351 9570 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:03:44.406083 9570 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:03:44.406399 9570 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:03:44.949515 6963 tablet_server.cc:178] TabletServer@127.6.204.193:0 shutting down...
I20240502 08:03:44.973461 6963 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 08:03:44.974174 6963 tablet_replica.cc:330] T f85e880e71c54e72a5218f2f058e1848 P 62f75069b443426690d0fbc60773d0b6: stopping tablet replica
I20240502 08:03:44.974731 6963 raft_consensus.cc:2229] T f85e880e71c54e72a5218f2f058e1848 P 62f75069b443426690d0fbc60773d0b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:03:44.975282 6963 raft_consensus.cc:2258] T f85e880e71c54e72a5218f2f058e1848 P 62f75069b443426690d0fbc60773d0b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:03:44.995191 6963 tablet_server.cc:194] TabletServer@127.6.204.193:0 shutdown complete.
I20240502 08:03:45.006987 6963 tablet_server.cc:178] TabletServer@127.6.204.194:0 shutting down...
I20240502 08:03:45.030586 6963 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 08:03:45.031340 6963 tablet_replica.cc:330] T f85e880e71c54e72a5218f2f058e1848 P 4d753a6ad0e5457d91fadd5c694cdd57: stopping tablet replica
I20240502 08:03:45.031939 6963 raft_consensus.cc:2229] T f85e880e71c54e72a5218f2f058e1848 P 4d753a6ad0e5457d91fadd5c694cdd57 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:03:45.032441 6963 raft_consensus.cc:2258] T f85e880e71c54e72a5218f2f058e1848 P 4d753a6ad0e5457d91fadd5c694cdd57 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240502 08:03:45.041510 9807 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.204.193:33309: connect: Connection refused (error 111) [suppressed 2 similar messages]
W20240502 08:03:45.044445 9807 consensus_peers.cc:485] T f85e880e71c54e72a5218f2f058e1848 P 763aeeacf0c74a82a9fdc368b40e9cd4 -> Peer 62f75069b443426690d0fbc60773d0b6 (127.6.204.193:33309): Couldn't send request to peer 62f75069b443426690d0fbc60773d0b6. Status: Network error: Client connection negotiation failed: client connection to 127.6.204.193:33309: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:03:45.051542 6963 tablet_server.cc:194] TabletServer@127.6.204.194:0 shutdown complete.
I20240502 08:03:45.061450 6963 tablet_server.cc:178] TabletServer@127.6.204.195:0 shutting down...
I20240502 08:03:45.082885 6963 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 08:03:45.098814 6963 tablet_server.cc:194] TabletServer@127.6.204.195:0 shutdown complete.
I20240502 08:03:45.107594 6963 tablet_server.cc:178] TabletServer@127.6.204.196:0 shutting down...
I20240502 08:03:45.130596 6963 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 08:03:45.131309 6963 tablet_replica.cc:330] T f85e880e71c54e72a5218f2f058e1848 P 763aeeacf0c74a82a9fdc368b40e9cd4: stopping tablet replica
I20240502 08:03:45.131850 6963 raft_consensus.cc:2229] T f85e880e71c54e72a5218f2f058e1848 P 763aeeacf0c74a82a9fdc368b40e9cd4 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:03:45.132683 6963 raft_consensus.cc:2258] T f85e880e71c54e72a5218f2f058e1848 P 763aeeacf0c74a82a9fdc368b40e9cd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:03:45.151554 6963 tablet_server.cc:194] TabletServer@127.6.204.196:0 shutdown complete.
I20240502 08:03:45.161876 6963 master.cc:537] Master@127.6.204.254:43021 shutting down...
I20240502 08:03:45.179122 6963 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 36815569922d42b594948340ac5b11df [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:03:45.179683 6963 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 36815569922d42b594948340ac5b11df [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:03:45.179991 6963 tablet_replica.cc:330] T 00000000000000000000000000000000 P 36815569922d42b594948340ac5b11df: stopping tablet replica
I20240502 08:03:45.197599 6963 master.cc:552] Master@127.6.204.254:43021 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne (12975 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy
I20240502 08:03:45.231606 6963 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.6.204.254:44097
I20240502 08:03:45.232710 6963 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:03:45.238433 9913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:03:45.238809 9914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:03:45.239485 9916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:03:45.240746 6963 server_base.cc:1034] running on GCE node
I20240502 08:03:45.241588 6963 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:03:45.241784 6963 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:03:45.241933 6963 hybrid_clock.cc:648] HybridClock initialized: now 1714637025241914 us; error 0 us; skew 500 ppm
I20240502 08:03:45.242468 6963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:03:45.244848 6963 webserver.cc:460] Webserver started at http://127.6.204.254:34081/ using document root <none> and password file <none>
I20240502 08:03:45.245324 6963 fs_manager.cc:359] Metadata directory not provided
I20240502 08:03:45.245491 6963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:03:45.245767 6963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:03:45.247016 6963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714636933625983-6963-0/minicluster-data/master-0-root/instance:
uuid: "df2018f1d8454a4c94bec17e37b51db0"
format_stamp: "Formatted at 2024-05-02 08:03:45 on dist-test-slave-32xs"
I20240502 08:03:45.251394 6963 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 08:03:45.254575 9921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:03:45.255290 6963 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 08:03:45.255550 6963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714636933625983-6963-0/minicluster-data/master-0-root
uuid: "df2018f1d8454a4c94bec17e37b51db0"
format_stamp: "Formatted at 2024-05-02 08:03:45 on dist-test-slave-32xs"
I20240502 08:03:45.255822 6963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714636933625983-6963-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714636933625983-6963-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714636933625983-6963-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:03:45.284920 6963 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:03:45.286151 6963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:03:45.323856 6963 rpc_server.cc:307] RPC server started. Bound to: 127.6.204.254:44097
I20240502 08:03:45.324003 9972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.204.254:44097 every 8 connection(s)
I20240502 08:03:45.327754 9973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:03:45.338724 9973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df2018f1d8454a4c94bec17e37b51db0: Bootstrap starting.
I20240502 08:03:45.343166 9973 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df2018f1d8454a4c94bec17e37b51db0: Neither blocks nor log segments found. Creating new log.
I20240502 08:03:45.347368 9973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df2018f1d8454a4c94bec17e37b51db0: No bootstrap required, opened a new log
I20240502 08:03:45.349424 9973 raft_consensus.cc:348] T 00000000000000000000000000000000 P df2018f1d8454a4c94bec17e37b51db0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df2018f1d8454a4c94bec17e37b51db0" member_type: VOTER }
I20240502 08:03:45.349917 9973 raft_consensus.cc:374] T 00000000000000000000000000000000 P df2018f1d8454a4c94bec17e37b51db0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:03:45.350111 9973 raft_consensus.cc:729] T 00000000000000000000000000000000 P df2018f1d8454a4c94bec17e37b51db0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df2018f1d8454a4c94bec17e37b51db0, State: Initialized, Role: FOLLOWER
I20240502 08:03:45.350677 9973 consensus_queue.cc:260] T 00000000000000000000000000000000 P df2018f1d8454a4c94bec17e37b51db0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df2018f1d8454a4c94bec17e37b51db0" member_type: VOTER }
I20240502 08:03:45.351155 9973 raft_consensus.cc:388] T 00000000000000000000000000000000 P df2018f1d8454a4c94bec17e37b51db0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:03:45.351356 9973 raft_consensus.cc:482] T 00000000000000000000000000000000 P df2018f1d8454a4c94bec17e37b51db0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:03:45.351585 9973 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df2018f1d8454a4c94bec17e37b51db0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:03:45.355971 9973 raft_consensus.cc:504] T 00000000000000000000000000000000 P df2018f1d8454a4c94bec17e37b51db0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df2018f1d8454a4c94bec17e37b51db0" member_type: VOTER }
I20240502 08:03:45.356494 9973 leader_election.cc:302] T 00000000000000000000000000000000 P df2018f1d8454a4c94bec17e37b51db0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df2018f1d8454a4c94bec17e37b51db0; no voters:
I20240502 08:03:45.357740 9973 leader_election.cc:288] T 00000000000000000000000000000000 P df2018f1d8454a4c94bec17e37b51db0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:03:45.358124 9976 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df2018f1d8454a4c94bec17e37b51db0 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:03:45.359434 9976 raft_consensus.cc:686] T 00000000000000000000000000000000 P df2018f1d8454a4c94bec17e37b51db0 [term 1 LEADER]: Becoming Leader. State: Replica: df2018f1d8454a4c94bec17e37b51db0, State: Running, Role: LEADER
I20240502 08:03:45.360126 9976 consensus_queue.cc:237] T 00000000000000000000000000000000 P df2018f1d8454a4c94bec17e37b51db0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df2018f1d8454a4c94bec17e37b51db0" member_type: VOTER }
I20240502 08:03:45.360849 9973 sys_catalog.cc:564] T 00000000000000000000000000000000 P df2018f1d8454a4c94bec17e37b51db0 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 08:03:45.363358 9978 sys_catalog.cc:455] T 00000000000000000000000000000000 P df2018f1d8454a4c94bec17e37b51db0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader df2018f1d8454a4c94bec17e37b51db0. Latest consensus state: current_term: 1 leader_uuid: "df2018f1d8454a4c94bec17e37b51db0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df2018f1d8454a4c94bec17e37b51db0" member_type: VOTER } }
I20240502 08:03:45.363339 9977 sys_catalog.cc:455] T 00000000000000000000000000000000 P df2018f1d8454a4c94bec17e37b51db0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df2018f1d8454a4c94bec17e37b51db0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df2018f1d8454a4c94bec17e37b51db0" member_type: VOTER } }
I20240502 08:03:45.364102 9978 sys_catalog.cc:458] T 00000000000000000000000000000000 P df2018f1d8454a4c94bec17e37b51db0 [sys.catalog]: This master's current role is: LEADER
I20240502 08:03:45.364264 9977 sys_catalog.cc:458] T 00000000000000000000000000000000 P df2018f1d8454a4c94bec17e37b51db0 [sys.catalog]: This master's current role is: LEADER
I20240502 08:03:45.368069 9982 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 08:03:45.374504 9982 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 08:03:45.376111 6963 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 08:03:45.386255 9982 catalog_manager.cc:1347] Generated new cluster ID: d3dc2205cec8482fbbf045f56b5bf065
I20240502 08:03:45.386464 9982 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 08:03:45.411909 9982 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 08:03:45.413134 9982 catalog_manager.cc:1504] Loading token signing keys...
I20240502 08:03:45.430249 9982 catalog_manager.cc:5857] T 00000000000000000000000000000000 P df2018f1d8454a4c94bec17e37b51db0: Generated new TSK 0
I20240502 08:03:45.430824 9982 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 08:03:45.443070 6963 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:03:45.448516 9994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:03:45.449735 9995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:03:45.451582 6963 server_base.cc:1034] running on GCE node
W20240502 08:03:45.451761 9997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:03:45.452556 6963 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:03:45.452749 6963 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:03:45.452913 6963 hybrid_clock.cc:648] HybridClock initialized: now 1714637025452893 us; error 0 us; skew 500 ppm
I20240502 08:03:45.453403 6963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:03:45.456308 6963 webserver.cc:460] Webserver started at http://127.6.204.193:37057/ using document root <none> and password file <none>
I20240502 08:03:45.456779 6963 fs_manager.cc:359] Metadata directory not provided
I20240502 08:03:45.456943 6963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:03:45.457196 6963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:03:45.458380 6963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714636933625983-6963-0/minicluster-data/ts-0-root/instance:
uuid: "56a79b0d441b4c9ebbf31d47fdefcb20"
format_stamp: "Formatted at 2024-05-02 08:03:45 on dist-test-slave-32xs"
I20240502 08:03:45.462783 6963 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240502 08:03:45.465865 10002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:03:45.466629 6963 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 08:03:45.466907 6963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714636933625983-6963-0/minicluster-data/ts-0-root
uuid: "56a79b0d441b4c9ebbf31d47fdefcb20"
format_stamp: "Formatted at 2024-05-02 08:03:45 on dist-test-slave-32xs"
I20240502 08:03:45.467180 6963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714636933625983-6963-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714636933625983-6963-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714636933625983-6963-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:03:45.481102 6963 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:03:45.482249 6963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:03:45.483664 6963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:03:45.485920 6963 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:03:45.486119 6963 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:03:45.486389 6963 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:03:45.486557 6963 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:03:45.525842 6963 rpc_server.cc:307] RPC server started. Bound to: 127.6.204.193:39795
I20240502 08:03:45.525938 10064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.204.193:39795 every 8 connection(s)
I20240502 08:03:45.530325 6963 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:03:45.538892 10069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:03:45.540148 10070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:03:45.543398 10072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:03:45.544896 6963 server_base.cc:1034] running on GCE node
I20240502 08:03:45.545540 10065 heartbeater.cc:347] Connected to a master server at 127.6.204.254:44097
I20240502 08:03:45.545773 6963 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20240502 08:03:45.545853 10065 heartbeater.cc:464] Registering TS with master...
W20240502 08:03:45.546021 6963 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:03:45.546274 6963 hybrid_clock.cc:648] HybridClock initialized: now 1714637025546256 us; error 0 us; skew 500 ppm
I20240502 08:03:45.546720 10065 heartbeater.cc:511] Master 127.6.204.254:44097 requested a full tablet report, sending...
I20240502 08:03:45.546828 6963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:03:45.548756 9938 ts_manager.cc:194] Registered new tserver with Master: 56a79b0d441b4c9ebbf31d47fdefcb20 (127.6.204.193:39795)
I20240502 08:03:45.549546 6963 webserver.cc:460] Webserver started at http://127.6.204.194:40443/ using document root <none> and password file <none>
I20240502 08:03:45.549996 6963 fs_manager.cc:359] Metadata directory not provided
I20240502 08:03:45.550199 6963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:03:45.550449 6963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:03:45.550652 9938 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60834
I20240502 08:03:45.551728 6963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714636933625983-6963-0/minicluster-data/ts-1-root/instance:
uuid: "a366f6a093a94d1fa011ca0746fd959a"
format_stamp: "Formatted at 2024-05-02 08:03:45 on dist-test-slave-32xs"
I20240502 08:03:45.556011 6963 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20240502 08:03:45.559341 10077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:03:45.560011 6963 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 08:03:45.560289 6963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714636933625983-6963-0/minicluster-data/ts-1-root
uuid: "a366f6a093a94d1fa011ca0746fd959a"
format_stamp: "Formatted at 2024-05-02 08:03:45 on dist-test-slave-32xs"
I20240502 08:03:45.560550 6963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714636933625983-6963-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714636933625983-6963-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714636933625983-6963-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:03:45.574448 6963 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:03:45.575531 6963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:03:45.576862 6963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:03:45.579331 6963 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:03:45.579588 6963 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:03:45.579859 6963 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:03:45.580086 6963 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:03:45.619117 6963 rpc_server.cc:307] RPC server started. Bound to: 127.6.204.194:42217
I20240502 08:03:45.619217 10139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.204.194:42217 every 8 connection(s)
I20240502 08:03:45.623729 6963 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:03:45.630337 10143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:03:45.633262 10144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:03:45.635080 10146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:03:45.635370 6963 server_base.cc:1034] running on GCE node
I20240502 08:03:45.635994 10140 heartbeater.cc:347] Connected to a master server at 127.6.204.254:44097
I20240502 08:03:45.636417 10140 heartbeater.cc:464] Registering TS with master...
I20240502 08:03:45.636605 6963 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:03:45.636869 6963 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:03:45.637075 6963 hybrid_clock.cc:648] HybridClock initialized: now 1714637025637056 us; error 0 us; skew 500 ppm
I20240502 08:03:45.637295 10140 heartbeater.cc:511] Master 127.6.204.254:44097 requested a full tablet report, sending...
I20240502 08:03:45.637802 6963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:03:45.639546 9938 ts_manager.cc:194] Registered new tserver with Master: a366f6a093a94d1fa011ca0746fd959a (127.6.204.194:42217)
I20240502 08:03:45.640540 6963 webserver.cc:460] Webserver started at http://127.6.204.195:35705/ using document root <none> and password file <none>
I20240502 08:03:45.641009 6963 fs_manager.cc:359] Metadata directory not provided
I20240502 08:03:45.641036 9938 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60842
I20240502 08:03:45.641258 6963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:03:45.641577 6963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:03:45.642882 6963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714636933625983-6963-0/minicluster-data/ts-2-root/instance:
uuid: "be0ceb65444347e28b47e1535ffeb29d"
format_stamp: "Formatted at 2024-05-02 08:03:45 on dist-test-slave-32xs"
I20240502 08:03:45.647038 6963 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20240502 08:03:45.650003 10151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:03:45.650738 6963 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 08:03:45.651002 6963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714636933625983-6963-0/minicluster-data/ts-2-root
uuid: "be0ceb65444347e28b47e1535ffeb29d"
format_stamp: "Formatted at 2024-05-02 08:03:45 on dist-test-slave-32xs"
I20240502 08:03:45.651247 6963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714636933625983-6963-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714636933625983-6963-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714636933625983-6963-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:03:45.663295 6963 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:03:45.664355 6963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:03:45.665746 6963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:03:45.668184 6963 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:03:45.668401 6963 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:03:45.668638 6963 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:03:45.668783 6963 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:03:45.707368 6963 rpc_server.cc:307] RPC server started. Bound to: 127.6.204.195:40491
I20240502 08:03:45.707489 10213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.204.195:40491 every 8 connection(s)
I20240502 08:03:45.720221 10214 heartbeater.cc:347] Connected to a master server at 127.6.204.254:44097
I20240502 08:03:45.720580 10214 heartbeater.cc:464] Registering TS with master...
I20240502 08:03:45.721239 10214 heartbeater.cc:511] Master 127.6.204.254:44097 requested a full tablet report, sending...
I20240502 08:03:45.723075 9938 ts_manager.cc:194] Registered new tserver with Master: be0ceb65444347e28b47e1535ffeb29d (127.6.204.195:40491)
I20240502 08:03:45.723928 6963 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013359104s
I20240502 08:03:45.724431 9938 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60846
I20240502 08:03:46.553018 10065 heartbeater.cc:503] Master 127.6.204.254:44097 was elected leader, sending a full tablet report...
I20240502 08:03:46.643406 10140 heartbeater.cc:503] Master 127.6.204.254:44097 was elected leader, sending a full tablet report...
I20240502 08:03:46.726855 10214 heartbeater.cc:503] Master 127.6.204.254:44097 was elected leader, sending a full tablet report...
I20240502 08:03:46.767689 6963 test_util.cc:268] Using random seed: 182858546
I20240502 08:03:46.787853 9938 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60876:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:03:46.789657 9938 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:03:46.839073 10179 tablet_service.cc:1467] Processing CreateTablet for tablet e052fad4dfb14df18ff81e27346aefa2 (DEFAULT_TABLE table=test-workload [id=031fe7e036754a6abf083ad9823a9fe1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:03:46.840456 10179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e052fad4dfb14df18ff81e27346aefa2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:03:46.844884 10105 tablet_service.cc:1467] Processing CreateTablet for tablet e052fad4dfb14df18ff81e27346aefa2 (DEFAULT_TABLE table=test-workload [id=031fe7e036754a6abf083ad9823a9fe1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:03:46.845187 10030 tablet_service.cc:1467] Processing CreateTablet for tablet e052fad4dfb14df18ff81e27346aefa2 (DEFAULT_TABLE table=test-workload [id=031fe7e036754a6abf083ad9823a9fe1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:03:46.845922 10105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e052fad4dfb14df18ff81e27346aefa2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:03:46.846447 10030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e052fad4dfb14df18ff81e27346aefa2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:03:46.857872 10235 tablet_bootstrap.cc:492] T e052fad4dfb14df18ff81e27346aefa2 P be0ceb65444347e28b47e1535ffeb29d: Bootstrap starting.
I20240502 08:03:46.859239 10236 tablet_bootstrap.cc:492] T e052fad4dfb14df18ff81e27346aefa2 P a366f6a093a94d1fa011ca0746fd959a: Bootstrap starting.
I20240502 08:03:46.863317 10237 tablet_bootstrap.cc:492] T e052fad4dfb14df18ff81e27346aefa2 P 56a79b0d441b4c9ebbf31d47fdefcb20: Bootstrap starting.
I20240502 08:03:46.865402 10236 tablet_bootstrap.cc:654] T e052fad4dfb14df18ff81e27346aefa2 P a366f6a093a94d1fa011ca0746fd959a: Neither blocks nor log segments found. Creating new log.
I20240502 08:03:46.866295 10235 tablet_bootstrap.cc:654] T e052fad4dfb14df18ff81e27346aefa2 P be0ceb65444347e28b47e1535ffeb29d: Neither blocks nor log segments found. Creating new log.
I20240502 08:03:46.869729 10237 tablet_bootstrap.cc:654] T e052fad4dfb14df18ff81e27346aefa2 P 56a79b0d441b4c9ebbf31d47fdefcb20: Neither blocks nor log segments found. Creating new log.
I20240502 08:03:46.870668 10236 tablet_bootstrap.cc:492] T e052fad4dfb14df18ff81e27346aefa2 P a366f6a093a94d1fa011ca0746fd959a: No bootstrap required, opened a new log
I20240502 08:03:46.871162 10236 ts_tablet_manager.cc:1367] T e052fad4dfb14df18ff81e27346aefa2 P a366f6a093a94d1fa011ca0746fd959a: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.004s
I20240502 08:03:46.873028 10235 tablet_bootstrap.cc:492] T e052fad4dfb14df18ff81e27346aefa2 P be0ceb65444347e28b47e1535ffeb29d: No bootstrap required, opened a new log
I20240502 08:03:46.873919 10235 ts_tablet_manager.cc:1367] T e052fad4dfb14df18ff81e27346aefa2 P be0ceb65444347e28b47e1535ffeb29d: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.004s
I20240502 08:03:46.873734 10236 raft_consensus.cc:348] T e052fad4dfb14df18ff81e27346aefa2 P a366f6a093a94d1fa011ca0746fd959a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a366f6a093a94d1fa011ca0746fd959a" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 42217 } } peers { permanent_uuid: "be0ceb65444347e28b47e1535ffeb29d" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40491 } } peers { permanent_uuid: "56a79b0d441b4c9ebbf31d47fdefcb20" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39795 } }
I20240502 08:03:46.874580 10236 raft_consensus.cc:374] T e052fad4dfb14df18ff81e27346aefa2 P a366f6a093a94d1fa011ca0746fd959a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:03:46.874977 10236 raft_consensus.cc:729] T e052fad4dfb14df18ff81e27346aefa2 P a366f6a093a94d1fa011ca0746fd959a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a366f6a093a94d1fa011ca0746fd959a, State: Initialized, Role: FOLLOWER
I20240502 08:03:46.875692 10236 consensus_queue.cc:260] T e052fad4dfb14df18ff81e27346aefa2 P a366f6a093a94d1fa011ca0746fd959a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a366f6a093a94d1fa011ca0746fd959a" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 42217 } } peers { permanent_uuid: "be0ceb65444347e28b47e1535ffeb29d" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40491 } } peers { permanent_uuid: "56a79b0d441b4c9ebbf31d47fdefcb20" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39795 } }
I20240502 08:03:46.876753 10237 tablet_bootstrap.cc:492] T e052fad4dfb14df18ff81e27346aefa2 P 56a79b0d441b4c9ebbf31d47fdefcb20: No bootstrap required, opened a new log
I20240502 08:03:46.876801 10235 raft_consensus.cc:348] T e052fad4dfb14df18ff81e27346aefa2 P be0ceb65444347e28b47e1535ffeb29d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a366f6a093a94d1fa011ca0746fd959a" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 42217 } } peers { permanent_uuid: "be0ceb65444347e28b47e1535ffeb29d" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40491 } } peers { permanent_uuid: "56a79b0d441b4c9ebbf31d47fdefcb20" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39795 } }
I20240502 08:03:46.877398 10237 ts_tablet_manager.cc:1367] T e052fad4dfb14df18ff81e27346aefa2 P 56a79b0d441b4c9ebbf31d47fdefcb20: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.006s
I20240502 08:03:46.877720 10235 raft_consensus.cc:374] T e052fad4dfb14df18ff81e27346aefa2 P be0ceb65444347e28b47e1535ffeb29d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:03:46.878074 10235 raft_consensus.cc:729] T e052fad4dfb14df18ff81e27346aefa2 P be0ceb65444347e28b47e1535ffeb29d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be0ceb65444347e28b47e1535ffeb29d, State: Initialized, Role: FOLLOWER
I20240502 08:03:46.878334 10236 ts_tablet_manager.cc:1398] T e052fad4dfb14df18ff81e27346aefa2 P a366f6a093a94d1fa011ca0746fd959a: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20240502 08:03:46.878847 10235 consensus_queue.cc:260] T e052fad4dfb14df18ff81e27346aefa2 P be0ceb65444347e28b47e1535ffeb29d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a366f6a093a94d1fa011ca0746fd959a" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 42217 } } peers { permanent_uuid: "be0ceb65444347e28b47e1535ffeb29d" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40491 } } peers { permanent_uuid: "56a79b0d441b4c9ebbf31d47fdefcb20" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39795 } }
I20240502 08:03:46.881814 10235 ts_tablet_manager.cc:1398] T e052fad4dfb14df18ff81e27346aefa2 P be0ceb65444347e28b47e1535ffeb29d: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240502 08:03:46.880625 10237 raft_consensus.cc:348] T e052fad4dfb14df18ff81e27346aefa2 P 56a79b0d441b4c9ebbf31d47fdefcb20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a366f6a093a94d1fa011ca0746fd959a" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 42217 } } peers { permanent_uuid: "be0ceb65444347e28b47e1535ffeb29d" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40491 } } peers { permanent_uuid: "56a79b0d441b4c9ebbf31d47fdefcb20" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39795 } }
I20240502 08:03:46.882629 10237 raft_consensus.cc:374] T e052fad4dfb14df18ff81e27346aefa2 P 56a79b0d441b4c9ebbf31d47fdefcb20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:03:46.882934 10237 raft_consensus.cc:729] T e052fad4dfb14df18ff81e27346aefa2 P 56a79b0d441b4c9ebbf31d47fdefcb20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56a79b0d441b4c9ebbf31d47fdefcb20, State: Initialized, Role: FOLLOWER
I20240502 08:03:46.883631 10237 consensus_queue.cc:260] T e052fad4dfb14df18ff81e27346aefa2 P 56a79b0d441b4c9ebbf31d47fdefcb20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a366f6a093a94d1fa011ca0746fd959a" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 42217 } } peers { permanent_uuid: "be0ceb65444347e28b47e1535ffeb29d" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40491 } } peers { permanent_uuid: "56a79b0d441b4c9ebbf31d47fdefcb20" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39795 } }
I20240502 08:03:46.889660 10237 ts_tablet_manager.cc:1398] T e052fad4dfb14df18ff81e27346aefa2 P 56a79b0d441b4c9ebbf31d47fdefcb20: Time spent starting tablet: real 0.012s user 0.004s sys 0.005s
I20240502 08:03:47.045512 10243 raft_consensus.cc:482] T e052fad4dfb14df18ff81e27346aefa2 P 56a79b0d441b4c9ebbf31d47fdefcb20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:03:47.045967 10243 raft_consensus.cc:504] T e052fad4dfb14df18ff81e27346aefa2 P 56a79b0d441b4c9ebbf31d47fdefcb20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a366f6a093a94d1fa011ca0746fd959a" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 42217 } } peers { permanent_uuid: "be0ceb65444347e28b47e1535ffeb29d" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40491 } } peers { permanent_uuid: "56a79b0d441b4c9ebbf31d47fdefcb20" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39795 } }
I20240502 08:03:47.048067 10243 leader_election.cc:288] T e052fad4dfb14df18ff81e27346aefa2 P 56a79b0d441b4c9ebbf31d47fdefcb20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a366f6a093a94d1fa011ca0746fd959a (127.6.204.194:42217), be0ceb65444347e28b47e1535ffeb29d (127.6.204.195:40491)
I20240502 08:03:47.058578 10115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e052fad4dfb14df18ff81e27346aefa2" candidate_uuid: "56a79b0d441b4c9ebbf31d47fdefcb20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a366f6a093a94d1fa011ca0746fd959a" is_pre_election: true
I20240502 08:03:47.058575 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e052fad4dfb14df18ff81e27346aefa2" candidate_uuid: "56a79b0d441b4c9ebbf31d47fdefcb20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be0ceb65444347e28b47e1535ffeb29d" is_pre_election: true
I20240502 08:03:47.059407 10115 raft_consensus.cc:2454] T e052fad4dfb14df18ff81e27346aefa2 P a366f6a093a94d1fa011ca0746fd959a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56a79b0d441b4c9ebbf31d47fdefcb20 in term 0.
I20240502 08:03:47.059468 10189 raft_consensus.cc:2454] T e052fad4dfb14df18ff81e27346aefa2 P be0ceb65444347e28b47e1535ffeb29d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56a79b0d441b4c9ebbf31d47fdefcb20 in term 0.
I20240502 08:03:47.060562 10006 leader_election.cc:302] T e052fad4dfb14df18ff81e27346aefa2 P 56a79b0d441b4c9ebbf31d47fdefcb20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 56a79b0d441b4c9ebbf31d47fdefcb20, a366f6a093a94d1fa011ca0746fd959a, be0ceb65444347e28b47e1535ffeb29d; no voters:
I20240502 08:03:47.061264 10243 raft_consensus.cc:2781] T e052fad4dfb14df18ff81e27346aefa2 P 56a79b0d441b4c9ebbf31d47fdefcb20 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:03:47.061560 10243 raft_consensus.cc:482] T e052fad4dfb14df18ff81e27346aefa2 P 56a79b0d441b4c9ebbf31d47fdefcb20 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:03:47.061802 10243 raft_consensus.cc:3037] T e052fad4dfb14df18ff81e27346aefa2 P 56a79b0d441b4c9ebbf31d47fdefcb20 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:03:47.066887 10243 raft_consensus.cc:504] T e052fad4dfb14df18ff81e27346aefa2 P 56a79b0d441b4c9ebbf31d47fdefcb20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a366f6a093a94d1fa011ca0746fd959a" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 42217 } } peers { permanent_uuid: "be0ceb65444347e28b47e1535ffeb29d" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40491 } } peers { permanent_uuid: "56a79b0d441b4c9ebbf31d47fdefcb20" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39795 } }
I20240502 08:03:47.068559 10243 leader_election.cc:288] T e052fad4dfb14df18ff81e27346aefa2 P 56a79b0d441b4c9ebbf31d47fdefcb20 [CANDIDATE]: Term 1 election: Requested vote from peers a366f6a093a94d1fa011ca0746fd959a (127.6.204.194:42217), be0ceb65444347e28b47e1535ffeb29d (127.6.204.195:40491)
I20240502 08:03:47.069489 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e052fad4dfb14df18ff81e27346aefa2" candidate_uuid: "56a79b0d441b4c9ebbf31d47fdefcb20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be0ceb65444347e28b47e1535ffeb29d"
I20240502 08:03:47.069574 10115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e052fad4dfb14df18ff81e27346aefa2" candidate_uuid: "56a79b0d441b4c9ebbf31d47fdefcb20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a366f6a093a94d1fa011ca0746fd959a"
I20240502 08:03:47.070058 10189 raft_consensus.cc:3037] T e052fad4dfb14df18ff81e27346aefa2 P be0ceb65444347e28b47e1535ffeb29d [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:03:47.070106 10115 raft_consensus.cc:3037] T e052fad4dfb14df18ff81e27346aefa2 P a366f6a093a94d1fa011ca0746fd959a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:03:47.074913 10115 raft_consensus.cc:2454] T e052fad4dfb14df18ff81e27346aefa2 P a366f6a093a94d1fa011ca0746fd959a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56a79b0d441b4c9ebbf31d47fdefcb20 in term 1.
I20240502 08:03:47.074925 10189 raft_consensus.cc:2454] T e052fad4dfb14df18ff81e27346aefa2 P be0ceb65444347e28b47e1535ffeb29d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56a79b0d441b4c9ebbf31d47fdefcb20 in term 1.
I20240502 08:03:47.075896 10006 leader_election.cc:302] T e052fad4dfb14df18ff81e27346aefa2 P 56a79b0d441b4c9ebbf31d47fdefcb20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56a79b0d441b4c9ebbf31d47fdefcb20, a366f6a093a94d1fa011ca0746fd959a; no voters:
I20240502 08:03:47.076540 10243 raft_consensus.cc:2781] T e052fad4dfb14df18ff81e27346aefa2 P 56a79b0d441b4c9ebbf31d47fdefcb20 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:03:47.077538 10243 raft_consensus.cc:686] T e052fad4dfb14df18ff81e27346aefa2 P 56a79b0d441b4c9ebbf31d47fdefcb20 [term 1 LEADER]: Becoming Leader. State: Replica: 56a79b0d441b4c9ebbf31d47fdefcb20, State: Running, Role: LEADER
I20240502 08:03:47.078265 10243 consensus_queue.cc:237] T e052fad4dfb14df18ff81e27346aefa2 P 56a79b0d441b4c9ebbf31d47fdefcb20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a366f6a093a94d1fa011ca0746fd959a" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 42217 } } peers { permanent_uuid: "be0ceb65444347e28b47e1535ffeb29d" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40491 } } peers { permanent_uuid: "56a79b0d441b4c9ebbf31d47fdefcb20" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39795 } }
I20240502 08:03:47.084508 9937 catalog_manager.cc:5484] T e052fad4dfb14df18ff81e27346aefa2 P 56a79b0d441b4c9ebbf31d47fdefcb20 reported cstate change: term changed from 0 to 1, leader changed from <none> to 56a79b0d441b4c9ebbf31d47fdefcb20 (127.6.204.193). New cstate: current_term: 1 leader_uuid: "56a79b0d441b4c9ebbf31d47fdefcb20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a366f6a093a94d1fa011ca0746fd959a" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 42217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be0ceb65444347e28b47e1535ffeb29d" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56a79b0d441b4c9ebbf31d47fdefcb20" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39795 } health_report { overall_health: HEALTHY } } }
I20240502 08:03:47.373518 9986 placement_policy_util.cc:407] tablet e052fad4dfb14df18ff81e27346aefa2: detected majority of replicas (2 of 3) at location L0
I20240502 08:03:47.374882 9991 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:03:47.376418 9991 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:03:47.376734 9991 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:03:47.572152 10243 consensus_queue.cc:1035] T e052fad4dfb14df18ff81e27346aefa2 P 56a79b0d441b4c9ebbf31d47fdefcb20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a366f6a093a94d1fa011ca0746fd959a" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 42217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:03:47.585835 10250 consensus_queue.cc:1035] T e052fad4dfb14df18ff81e27346aefa2 P 56a79b0d441b4c9ebbf31d47fdefcb20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be0ceb65444347e28b47e1535ffeb29d" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 08:03:48.381944 9986 placement_policy_util.cc:407] tablet e052fad4dfb14df18ff81e27346aefa2: detected majority of replicas (2 of 3) at location L0
I20240502 08:03:49.377691 9991 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:03:49.379000 9991 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:03:49.379279 9991 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:03:49.389564 9986 placement_policy_util.cc:407] tablet e052fad4dfb14df18ff81e27346aefa2: detected majority of replicas (2 of 3) at location L0
I20240502 08:03:50.397138 9986 placement_policy_util.cc:407] tablet e052fad4dfb14df18ff81e27346aefa2: detected majority of replicas (2 of 3) at location L0
I20240502 08:03:51.380159 9991 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:03:51.381415 9991 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:03:51.381705 9991 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:03:51.404884 9986 placement_policy_util.cc:407] tablet e052fad4dfb14df18ff81e27346aefa2: detected majority of replicas (2 of 3) at location L0
I20240502 08:03:52.412552 9986 placement_policy_util.cc:407] tablet e052fad4dfb14df18ff81e27346aefa2: detected majority of replicas (2 of 3) at location L0
I20240502 08:03:53.382649 9991 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:03:53.384049 9991 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:03:53.384351 9991 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:03:53.422293 9986 placement_policy_util.cc:407] tablet e052fad4dfb14df18ff81e27346aefa2: detected majority of replicas (2 of 3) at location L0
I20240502 08:03:54.431850 9986 placement_policy_util.cc:407] tablet e052fad4dfb14df18ff81e27346aefa2: detected majority of replicas (2 of 3) at location L0
I20240502 08:03:55.385475 9991 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:03:55.386955 9991 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:03:55.387245 9991 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:03:55.439785 9986 placement_policy_util.cc:407] tablet e052fad4dfb14df18ff81e27346aefa2: detected majority of replicas (2 of 3) at location L0
I20240502 08:03:56.446564 9986 placement_policy_util.cc:407] tablet e052fad4dfb14df18ff81e27346aefa2: detected majority of replicas (2 of 3) at location L0
I20240502 08:03:57.388191 9991 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:03:57.389501 9991 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:03:57.389819 9991 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:03:57.453163 9986 placement_policy_util.cc:407] tablet e052fad4dfb14df18ff81e27346aefa2: detected majority of replicas (2 of 3) at location L0
I20240502 08:03:58.189421 6963 tablet_server.cc:178] TabletServer@127.6.204.193:0 shutting down...
I20240502 08:03:58.214466 6963 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 08:03:58.215157 6963 tablet_replica.cc:330] T e052fad4dfb14df18ff81e27346aefa2 P 56a79b0d441b4c9ebbf31d47fdefcb20: stopping tablet replica
I20240502 08:03:58.215761 6963 raft_consensus.cc:2229] T e052fad4dfb14df18ff81e27346aefa2 P 56a79b0d441b4c9ebbf31d47fdefcb20 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:03:58.216698 6963 raft_consensus.cc:2258] T e052fad4dfb14df18ff81e27346aefa2 P 56a79b0d441b4c9ebbf31d47fdefcb20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:03:58.236718 6963 tablet_server.cc:194] TabletServer@127.6.204.193:0 shutdown complete.
I20240502 08:03:58.248618 6963 tablet_server.cc:178] TabletServer@127.6.204.194:0 shutting down...
I20240502 08:03:58.271708 6963 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 08:03:58.272334 6963 tablet_replica.cc:330] T e052fad4dfb14df18ff81e27346aefa2 P a366f6a093a94d1fa011ca0746fd959a: stopping tablet replica
I20240502 08:03:58.272903 6963 raft_consensus.cc:2229] T e052fad4dfb14df18ff81e27346aefa2 P a366f6a093a94d1fa011ca0746fd959a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:03:58.273361 6963 raft_consensus.cc:2258] T e052fad4dfb14df18ff81e27346aefa2 P a366f6a093a94d1fa011ca0746fd959a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:03:58.292367 6963 tablet_server.cc:194] TabletServer@127.6.204.194:0 shutdown complete.
I20240502 08:03:58.303673 6963 tablet_server.cc:178] TabletServer@127.6.204.195:0 shutting down...
I20240502 08:03:58.325868 6963 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 08:03:58.326622 6963 tablet_replica.cc:330] T e052fad4dfb14df18ff81e27346aefa2 P be0ceb65444347e28b47e1535ffeb29d: stopping tablet replica
I20240502 08:03:58.327126 6963 raft_consensus.cc:2229] T e052fad4dfb14df18ff81e27346aefa2 P be0ceb65444347e28b47e1535ffeb29d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:03:58.327584 6963 raft_consensus.cc:2258] T e052fad4dfb14df18ff81e27346aefa2 P be0ceb65444347e28b47e1535ffeb29d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:03:58.347285 6963 tablet_server.cc:194] TabletServer@127.6.204.195:0 shutdown complete.
I20240502 08:03:58.359179 6963 master.cc:537] Master@127.6.204.254:44097 shutting down...
I20240502 08:03:58.381922 6963 raft_consensus.cc:2229] T 00000000000000000000000000000000 P df2018f1d8454a4c94bec17e37b51db0 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:03:58.382612 6963 raft_consensus.cc:2258] T 00000000000000000000000000000000 P df2018f1d8454a4c94bec17e37b51db0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:03:58.382970 6963 tablet_replica.cc:330] T 00000000000000000000000000000000 P df2018f1d8454a4c94bec17e37b51db0: stopping tablet replica
I20240502 08:03:58.401000 6963 master.cc:552] Master@127.6.204.254:44097 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy (13203 ms)
[ RUN ] AutoRebalancerTest.TestMaxMovesPerServer
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/master/auto_rebalancer-test.cc:516: Skipped
test is skipped; set KUDU_ALLOW_SLOW_TESTS=1 to run
[ SKIPPED ] AutoRebalancerTest.TestMaxMovesPerServer (6 ms)
[ RUN ] AutoRebalancerTest.AutoRebalancingUnstableCluster
I20240502 08:03:58.440806 6963 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.6.204.254:46313
I20240502 08:03:58.441859 6963 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:03:58.447427 10259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:03:58.447846 10260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:03:58.449359 10262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:03:58.450503 6963 server_base.cc:1034] running on GCE node
I20240502 08:03:58.451288 6963 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:03:58.451465 6963 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:03:58.451629 6963 hybrid_clock.cc:648] HybridClock initialized: now 1714637038451605 us; error 0 us; skew 500 ppm
I20240502 08:03:58.452119 6963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:03:58.460168 6963 webserver.cc:460] Webserver started at http://127.6.204.254:42841/ using document root <none> and password file <none>
I20240502 08:03:58.460652 6963 fs_manager.cc:359] Metadata directory not provided
I20240502 08:03:58.460822 6963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:03:58.461117 6963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:03:58.462404 6963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714636933625983-6963-0/minicluster-data/master-0-root/instance:
uuid: "93264b44a7ca4f0f82ac003ffae0d042"
format_stamp: "Formatted at 2024-05-02 08:03:58 on dist-test-slave-32xs"
I20240502 08:03:58.466763 6963 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20240502 08:03:58.469930 10267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:03:58.470669 6963 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 08:03:58.470933 6963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714636933625983-6963-0/minicluster-data/master-0-root
uuid: "93264b44a7ca4f0f82ac003ffae0d042"
format_stamp: "Formatted at 2024-05-02 08:03:58 on dist-test-slave-32xs"
I20240502 08:03:58.471187 6963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714636933625983-6963-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714636933625983-6963-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714636933625983-6963-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:03:58.491328 6963 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:03:58.492477 6963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:03:58.529479 6963 rpc_server.cc:307] RPC server started. Bound to: 127.6.204.254:46313
I20240502 08:03:58.529575 10318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.204.254:46313 every 8 connection(s)
I20240502 08:03:58.533484 10319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:03:58.544955 10319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93264b44a7ca4f0f82ac003ffae0d042: Bootstrap starting.
I20240502 08:03:58.549229 10319 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93264b44a7ca4f0f82ac003ffae0d042: Neither blocks nor log segments found. Creating new log.
I20240502 08:03:58.553443 10319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93264b44a7ca4f0f82ac003ffae0d042: No bootstrap required, opened a new log
I20240502 08:03:58.555577 10319 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93264b44a7ca4f0f82ac003ffae0d042 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93264b44a7ca4f0f82ac003ffae0d042" member_type: VOTER }
I20240502 08:03:58.556044 10319 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93264b44a7ca4f0f82ac003ffae0d042 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:03:58.556223 10319 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93264b44a7ca4f0f82ac003ffae0d042 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93264b44a7ca4f0f82ac003ffae0d042, State: Initialized, Role: FOLLOWER
I20240502 08:03:58.556753 10319 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93264b44a7ca4f0f82ac003ffae0d042 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93264b44a7ca4f0f82ac003ffae0d042" member_type: VOTER }
I20240502 08:03:58.557219 10319 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93264b44a7ca4f0f82ac003ffae0d042 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:03:58.557404 10319 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93264b44a7ca4f0f82ac003ffae0d042 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:03:58.557632 10319 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93264b44a7ca4f0f82ac003ffae0d042 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:03:58.562109 10319 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93264b44a7ca4f0f82ac003ffae0d042 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93264b44a7ca4f0f82ac003ffae0d042" member_type: VOTER }
I20240502 08:03:58.562665 10319 leader_election.cc:302] T 00000000000000000000000000000000 P 93264b44a7ca4f0f82ac003ffae0d042 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93264b44a7ca4f0f82ac003ffae0d042; no voters:
I20240502 08:03:58.563719 10319 leader_election.cc:288] T 00000000000000000000000000000000 P 93264b44a7ca4f0f82ac003ffae0d042 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:03:58.564038 10322 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93264b44a7ca4f0f82ac003ffae0d042 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:03:58.565372 10322 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93264b44a7ca4f0f82ac003ffae0d042 [term 1 LEADER]: Becoming Leader. State: Replica: 93264b44a7ca4f0f82ac003ffae0d042, State: Running, Role: LEADER
I20240502 08:03:58.565977 10322 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93264b44a7ca4f0f82ac003ffae0d042 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93264b44a7ca4f0f82ac003ffae0d042" member_type: VOTER }
I20240502 08:03:58.566745 10319 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93264b44a7ca4f0f82ac003ffae0d042 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 08:03:58.568955 10324 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93264b44a7ca4f0f82ac003ffae0d042 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93264b44a7ca4f0f82ac003ffae0d042. Latest consensus state: current_term: 1 leader_uuid: "93264b44a7ca4f0f82ac003ffae0d042" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93264b44a7ca4f0f82ac003ffae0d042" member_type: VOTER } }
I20240502 08:03:58.569641 10324 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93264b44a7ca4f0f82ac003ffae0d042 [sys.catalog]: This master's current role is: LEADER
I20240502 08:03:58.568974 10323 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93264b44a7ca4f0f82ac003ffae0d042 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "93264b44a7ca4f0f82ac003ffae0d042" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93264b44a7ca4f0f82ac003ffae0d042" member_type: VOTER } }
I20240502 08:03:58.570830 10323 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93264b44a7ca4f0f82ac003ffae0d042 [sys.catalog]: This master's current role is: LEADER
I20240502 08:03:58.577250 10327 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 08:03:58.581883 10327 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 08:03:58.583540 6963 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 08:03:58.590117 10327 catalog_manager.cc:1347] Generated new cluster ID: eaddc36ac91f470190fb452ca5ec9741
I20240502 08:03:58.590420 10327 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 08:03:58.601267 10327 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 08:03:58.602592 10327 catalog_manager.cc:1504] Loading token signing keys...
I20240502 08:03:58.616195 10327 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 93264b44a7ca4f0f82ac003ffae0d042: Generated new TSK 0
I20240502 08:03:58.616815 10327 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 08:03:58.650015 6963 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:03:58.655771 10340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:03:58.656951 10341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:03:58.659729 10343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:03:58.660776 6963 server_base.cc:1034] running on GCE node
I20240502 08:03:58.661536 6963 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:03:58.661718 6963 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:03:58.661876 6963 hybrid_clock.cc:648] HybridClock initialized: now 1714637038661852 us; error 0 us; skew 500 ppm
I20240502 08:03:58.662449 6963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:03:58.664714 6963 webserver.cc:460] Webserver started at http://127.6.204.193:41215/ using document root <none> and password file <none>
I20240502 08:03:58.665150 6963 fs_manager.cc:359] Metadata directory not provided
I20240502 08:03:58.665330 6963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:03:58.665555 6963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:03:58.666867 6963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714636933625983-6963-0/minicluster-data/ts-0-root/instance:
uuid: "9416a559822d4196a77a162a8bb09b99"
format_stamp: "Formatted at 2024-05-02 08:03:58 on dist-test-slave-32xs"
I20240502 08:03:58.671186 6963 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240502 08:03:58.674371 10348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:03:58.675182 6963 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 08:03:58.675472 6963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714636933625983-6963-0/minicluster-data/ts-0-root
uuid: "9416a559822d4196a77a162a8bb09b99"
format_stamp: "Formatted at 2024-05-02 08:03:58 on dist-test-slave-32xs"
I20240502 08:03:58.675735 6963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714636933625983-6963-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714636933625983-6963-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714636933625983-6963-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:03:58.690100 6963 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:03:58.691182 6963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:03:58.692525 6963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:03:58.694766 6963 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:03:58.694953 6963 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:03:58.695178 6963 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:03:58.695341 6963 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:03:58.734717 6963 rpc_server.cc:307] RPC server started. Bound to: 127.6.204.193:34737
I20240502 08:03:58.734822 10410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.204.193:34737 every 8 connection(s)
I20240502 08:03:58.739151 6963 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:03:58.746727 10415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:03:58.749550 10416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:03:58.752588 10411 heartbeater.cc:347] Connected to a master server at 127.6.204.254:46313
I20240502 08:03:58.752653 6963 server_base.cc:1034] running on GCE node
I20240502 08:03:58.753120 10411 heartbeater.cc:464] Registering TS with master...
W20240502 08:03:58.753391 10418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:03:58.754029 10411 heartbeater.cc:511] Master 127.6.204.254:46313 requested a full tablet report, sending...
I20240502 08:03:58.754400 6963 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:03:58.754667 6963 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:03:58.754873 6963 hybrid_clock.cc:648] HybridClock initialized: now 1714637038754855 us; error 0 us; skew 500 ppm
I20240502 08:03:58.755613 6963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:03:58.756261 10284 ts_manager.cc:194] Registered new tserver with Master: 9416a559822d4196a77a162a8bb09b99 (127.6.204.193:34737)
I20240502 08:03:58.758303 10284 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41444
I20240502 08:03:58.758401 6963 webserver.cc:460] Webserver started at http://127.6.204.194:39707/ using document root <none> and password file <none>
I20240502 08:03:58.758955 6963 fs_manager.cc:359] Metadata directory not provided
I20240502 08:03:58.759124 6963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:03:58.759441 6963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:03:58.761019 6963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714636933625983-6963-0/minicluster-data/ts-1-root/instance:
uuid: "71013a2b500f4319bb04ffd933d71a8f"
format_stamp: "Formatted at 2024-05-02 08:03:58 on dist-test-slave-32xs"
I20240502 08:03:58.765202 6963 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20240502 08:03:58.768376 10423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:03:58.769133 6963 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240502 08:03:58.769385 6963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714636933625983-6963-0/minicluster-data/ts-1-root
uuid: "71013a2b500f4319bb04ffd933d71a8f"
format_stamp: "Formatted at 2024-05-02 08:03:58 on dist-test-slave-32xs"
I20240502 08:03:58.769644 6963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714636933625983-6963-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714636933625983-6963-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714636933625983-6963-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:03:58.781023 6963 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:03:58.782111 6963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:03:58.783469 6963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:03:58.785676 6963 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:03:58.785874 6963 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:03:58.786088 6963 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:03:58.786281 6963 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20240502 08:03:58.825294 6963 rpc_server.cc:307] RPC server started. Bound to: 127.6.204.194:42107
I20240502 08:03:58.825377 10485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.204.194:42107 every 8 connection(s)
I20240502 08:03:58.829946 6963 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:03:58.836491 10489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:03:58.838095 10490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:03:58.841508 10492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:03:58.842036 10486 heartbeater.cc:347] Connected to a master server at 127.6.204.254:46313
I20240502 08:03:58.842442 10486 heartbeater.cc:464] Registering TS with master...
I20240502 08:03:58.842396 6963 server_base.cc:1034] running on GCE node
I20240502 08:03:58.843369 10486 heartbeater.cc:511] Master 127.6.204.254:46313 requested a full tablet report, sending...
I20240502 08:03:58.843441 6963 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:03:58.843701 6963 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:03:58.843854 6963 hybrid_clock.cc:648] HybridClock initialized: now 1714637038843841 us; error 0 us; skew 500 ppm
I20240502 08:03:58.844575 6963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:03:58.845407 10284 ts_manager.cc:194] Registered new tserver with Master: 71013a2b500f4319bb04ffd933d71a8f (127.6.204.194:42107)
I20240502 08:03:58.847352 6963 webserver.cc:460] Webserver started at http://127.6.204.195:35355/ using document root <none> and password file <none>
I20240502 08:03:58.847471 10284 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41456
I20240502 08:03:58.847998 6963 fs_manager.cc:359] Metadata directory not provided
I20240502 08:03:58.848269 6963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:03:58.848562 6963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:03:58.850108 6963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714636933625983-6963-0/minicluster-data/ts-2-root/instance:
uuid: "9b42bd3f172344d59014088f13911462"
format_stamp: "Formatted at 2024-05-02 08:03:58 on dist-test-slave-32xs"
I20240502 08:03:58.854339 6963 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240502 08:03:58.857352 10497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:03:58.858059 6963 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 08:03:58.858350 6963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714636933625983-6963-0/minicluster-data/ts-2-root
uuid: "9b42bd3f172344d59014088f13911462"
format_stamp: "Formatted at 2024-05-02 08:03:58 on dist-test-slave-32xs"
I20240502 08:03:58.858584 6963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714636933625983-6963-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714636933625983-6963-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714636933625983-6963-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:03:58.877851 6963 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:03:58.879000 6963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:03:58.880398 6963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:03:58.882586 6963 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:03:58.882761 6963 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:03:58.883002 6963 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:03:58.883142 6963 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:03:58.922330 6963 rpc_server.cc:307] RPC server started. Bound to: 127.6.204.195:38051
I20240502 08:03:58.922405 10559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.204.195:38051 every 8 connection(s)
I20240502 08:03:58.936226 10560 heartbeater.cc:347] Connected to a master server at 127.6.204.254:46313
I20240502 08:03:58.936606 10560 heartbeater.cc:464] Registering TS with master...
I20240502 08:03:58.937363 10560 heartbeater.cc:511] Master 127.6.204.254:46313 requested a full tablet report, sending...
I20240502 08:03:58.939152 10284 ts_manager.cc:194] Registered new tserver with Master: 9b42bd3f172344d59014088f13911462 (127.6.204.195:38051)
I20240502 08:03:58.939338 6963 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013776246s
I20240502 08:03:58.941008 10284 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41466
I20240502 08:03:59.761065 10411 heartbeater.cc:503] Master 127.6.204.254:46313 was elected leader, sending a full tablet report...
I20240502 08:03:59.850268 10486 heartbeater.cc:503] Master 127.6.204.254:46313 was elected leader, sending a full tablet report...
I20240502 08:03:59.943931 10560 heartbeater.cc:503] Master 127.6.204.254:46313 was elected leader, sending a full tablet report...
I20240502 08:03:59.972505 6963 test_util.cc:268] Using random seed: 196063362
I20240502 08:03:59.994915 10284 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47366:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\252\252\252*\004\001\000TUUU""\004\001\000\252\252\252*\004\001\000TUUU"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:03:59.997215 10284 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:04:00.045712 10376 tablet_service.cc:1467] Processing CreateTablet for tablet 434c464e11b64a00ae9d3d3097ca345d (DEFAULT_TABLE table=test-workload [id=0640579302af4889b256057bbe62d105]), partition=RANGE (key) PARTITION VALUES < 715827882
I20240502 08:04:00.047221 10374 tablet_service.cc:1467] Processing CreateTablet for tablet 385fa1c24869499ab21a9a6852f82494 (DEFAULT_TABLE table=test-workload [id=0640579302af4889b256057bbe62d105]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20240502 08:04:00.047199 10376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 434c464e11b64a00ae9d3d3097ca345d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:00.047199 10451 tablet_service.cc:1467] Processing CreateTablet for tablet 434c464e11b64a00ae9d3d3097ca345d (DEFAULT_TABLE table=test-workload [id=0640579302af4889b256057bbe62d105]), partition=RANGE (key) PARTITION VALUES < 715827882
I20240502 08:04:00.049378 10375 tablet_service.cc:1467] Processing CreateTablet for tablet b69b32e4d5b04f00956ef81511e7c9eb (DEFAULT_TABLE table=test-workload [id=0640579302af4889b256057bbe62d105]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20240502 08:04:00.050742 10451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 434c464e11b64a00ae9d3d3097ca345d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:00.051281 10375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b69b32e4d5b04f00956ef81511e7c9eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:00.048676 10450 tablet_service.cc:1467] Processing CreateTablet for tablet b69b32e4d5b04f00956ef81511e7c9eb (DEFAULT_TABLE table=test-workload [id=0640579302af4889b256057bbe62d105]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20240502 08:04:00.054807 10374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 385fa1c24869499ab21a9a6852f82494. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:00.054967 10450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b69b32e4d5b04f00956ef81511e7c9eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:00.056447 10449 tablet_service.cc:1467] Processing CreateTablet for tablet 385fa1c24869499ab21a9a6852f82494 (DEFAULT_TABLE table=test-workload [id=0640579302af4889b256057bbe62d105]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20240502 08:04:00.057802 10449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 385fa1c24869499ab21a9a6852f82494. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:00.070528 10525 tablet_service.cc:1467] Processing CreateTablet for tablet 434c464e11b64a00ae9d3d3097ca345d (DEFAULT_TABLE table=test-workload [id=0640579302af4889b256057bbe62d105]), partition=RANGE (key) PARTITION VALUES < 715827882
I20240502 08:04:00.071986 10525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 434c464e11b64a00ae9d3d3097ca345d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:00.082055 10524 tablet_service.cc:1467] Processing CreateTablet for tablet b69b32e4d5b04f00956ef81511e7c9eb (DEFAULT_TABLE table=test-workload [id=0640579302af4889b256057bbe62d105]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20240502 08:04:00.083483 10524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b69b32e4d5b04f00956ef81511e7c9eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:00.091869 10523 tablet_service.cc:1467] Processing CreateTablet for tablet 385fa1c24869499ab21a9a6852f82494 (DEFAULT_TABLE table=test-workload [id=0640579302af4889b256057bbe62d105]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20240502 08:04:00.093299 10523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 385fa1c24869499ab21a9a6852f82494. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:00.096810 10580 tablet_bootstrap.cc:492] T 385fa1c24869499ab21a9a6852f82494 P 9416a559822d4196a77a162a8bb09b99: Bootstrap starting.
I20240502 08:04:00.102299 10582 tablet_bootstrap.cc:492] T 434c464e11b64a00ae9d3d3097ca345d P 9b42bd3f172344d59014088f13911462: Bootstrap starting.
I20240502 08:04:00.106261 10580 tablet_bootstrap.cc:654] T 385fa1c24869499ab21a9a6852f82494 P 9416a559822d4196a77a162a8bb09b99: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:00.102293 10581 tablet_bootstrap.cc:492] T b69b32e4d5b04f00956ef81511e7c9eb P 71013a2b500f4319bb04ffd933d71a8f: Bootstrap starting.
I20240502 08:04:00.110204 10582 tablet_bootstrap.cc:654] T 434c464e11b64a00ae9d3d3097ca345d P 9b42bd3f172344d59014088f13911462: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:00.118067 10582 tablet_bootstrap.cc:492] T 434c464e11b64a00ae9d3d3097ca345d P 9b42bd3f172344d59014088f13911462: No bootstrap required, opened a new log
I20240502 08:04:00.118499 10582 ts_tablet_manager.cc:1367] T 434c464e11b64a00ae9d3d3097ca345d P 9b42bd3f172344d59014088f13911462: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.007s
I20240502 08:04:00.120450 10582 raft_consensus.cc:348] T 434c464e11b64a00ae9d3d3097ca345d P 9b42bd3f172344d59014088f13911462 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416a559822d4196a77a162a8bb09b99" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 34737 } } peers { permanent_uuid: "71013a2b500f4319bb04ffd933d71a8f" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 42107 } } peers { permanent_uuid: "9b42bd3f172344d59014088f13911462" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38051 } }
I20240502 08:04:00.120973 10581 tablet_bootstrap.cc:654] T b69b32e4d5b04f00956ef81511e7c9eb P 71013a2b500f4319bb04ffd933d71a8f: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:00.121009 10582 raft_consensus.cc:374] T 434c464e11b64a00ae9d3d3097ca345d P 9b42bd3f172344d59014088f13911462 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:00.121649 10582 raft_consensus.cc:729] T 434c464e11b64a00ae9d3d3097ca345d P 9b42bd3f172344d59014088f13911462 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b42bd3f172344d59014088f13911462, State: Initialized, Role: FOLLOWER
I20240502 08:04:00.122467 10582 consensus_queue.cc:260] T 434c464e11b64a00ae9d3d3097ca345d P 9b42bd3f172344d59014088f13911462 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416a559822d4196a77a162a8bb09b99" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 34737 } } peers { permanent_uuid: "71013a2b500f4319bb04ffd933d71a8f" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 42107 } } peers { permanent_uuid: "9b42bd3f172344d59014088f13911462" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38051 } }
I20240502 08:04:00.123759 10580 tablet_bootstrap.cc:492] T 385fa1c24869499ab21a9a6852f82494 P 9416a559822d4196a77a162a8bb09b99: No bootstrap required, opened a new log
I20240502 08:04:00.124306 10580 ts_tablet_manager.cc:1367] T 385fa1c24869499ab21a9a6852f82494 P 9416a559822d4196a77a162a8bb09b99: Time spent bootstrapping tablet: real 0.028s user 0.009s sys 0.006s
I20240502 08:04:00.127009 10580 raft_consensus.cc:348] T 385fa1c24869499ab21a9a6852f82494 P 9416a559822d4196a77a162a8bb09b99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416a559822d4196a77a162a8bb09b99" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 34737 } } peers { permanent_uuid: "71013a2b500f4319bb04ffd933d71a8f" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 42107 } } peers { permanent_uuid: "9b42bd3f172344d59014088f13911462" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38051 } }
I20240502 08:04:00.127269 10581 tablet_bootstrap.cc:492] T b69b32e4d5b04f00956ef81511e7c9eb P 71013a2b500f4319bb04ffd933d71a8f: No bootstrap required, opened a new log
I20240502 08:04:00.127554 10580 raft_consensus.cc:374] T 385fa1c24869499ab21a9a6852f82494 P 9416a559822d4196a77a162a8bb09b99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:00.127879 10580 raft_consensus.cc:729] T 385fa1c24869499ab21a9a6852f82494 P 9416a559822d4196a77a162a8bb09b99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9416a559822d4196a77a162a8bb09b99, State: Initialized, Role: FOLLOWER
I20240502 08:04:00.127985 10581 ts_tablet_manager.cc:1367] T b69b32e4d5b04f00956ef81511e7c9eb P 71013a2b500f4319bb04ffd933d71a8f: Time spent bootstrapping tablet: real 0.027s user 0.010s sys 0.004s
I20240502 08:04:00.128463 10580 consensus_queue.cc:260] T 385fa1c24869499ab21a9a6852f82494 P 9416a559822d4196a77a162a8bb09b99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416a559822d4196a77a162a8bb09b99" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 34737 } } peers { permanent_uuid: "71013a2b500f4319bb04ffd933d71a8f" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 42107 } } peers { permanent_uuid: "9b42bd3f172344d59014088f13911462" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38051 } }
I20240502 08:04:00.131384 10580 ts_tablet_manager.cc:1398] T 385fa1c24869499ab21a9a6852f82494 P 9416a559822d4196a77a162a8bb09b99: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20240502 08:04:00.131609 10581 raft_consensus.cc:348] T b69b32e4d5b04f00956ef81511e7c9eb P 71013a2b500f4319bb04ffd933d71a8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416a559822d4196a77a162a8bb09b99" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 34737 } } peers { permanent_uuid: "71013a2b500f4319bb04ffd933d71a8f" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 42107 } } peers { permanent_uuid: "9b42bd3f172344d59014088f13911462" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38051 } }
I20240502 08:04:00.132477 10581 raft_consensus.cc:374] T b69b32e4d5b04f00956ef81511e7c9eb P 71013a2b500f4319bb04ffd933d71a8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:00.132489 10580 tablet_bootstrap.cc:492] T 434c464e11b64a00ae9d3d3097ca345d P 9416a559822d4196a77a162a8bb09b99: Bootstrap starting.
I20240502 08:04:00.132983 10581 raft_consensus.cc:729] T b69b32e4d5b04f00956ef81511e7c9eb P 71013a2b500f4319bb04ffd933d71a8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71013a2b500f4319bb04ffd933d71a8f, State: Initialized, Role: FOLLOWER
I20240502 08:04:00.132710 10582 ts_tablet_manager.cc:1398] T 434c464e11b64a00ae9d3d3097ca345d P 9b42bd3f172344d59014088f13911462: Time spent starting tablet: real 0.014s user 0.007s sys 0.004s
I20240502 08:04:00.133793 10581 consensus_queue.cc:260] T b69b32e4d5b04f00956ef81511e7c9eb P 71013a2b500f4319bb04ffd933d71a8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416a559822d4196a77a162a8bb09b99" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 34737 } } peers { permanent_uuid: "71013a2b500f4319bb04ffd933d71a8f" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 42107 } } peers { permanent_uuid: "9b42bd3f172344d59014088f13911462" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38051 } }
I20240502 08:04:00.135231 10582 tablet_bootstrap.cc:492] T 385fa1c24869499ab21a9a6852f82494 P 9b42bd3f172344d59014088f13911462: Bootstrap starting.
I20240502 08:04:00.141636 10582 tablet_bootstrap.cc:654] T 385fa1c24869499ab21a9a6852f82494 P 9b42bd3f172344d59014088f13911462: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:00.146934 10580 tablet_bootstrap.cc:654] T 434c464e11b64a00ae9d3d3097ca345d P 9416a559822d4196a77a162a8bb09b99: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:00.149729 10581 ts_tablet_manager.cc:1398] T b69b32e4d5b04f00956ef81511e7c9eb P 71013a2b500f4319bb04ffd933d71a8f: Time spent starting tablet: real 0.021s user 0.008s sys 0.012s
I20240502 08:04:00.150991 10581 tablet_bootstrap.cc:492] T 385fa1c24869499ab21a9a6852f82494 P 71013a2b500f4319bb04ffd933d71a8f: Bootstrap starting.
I20240502 08:04:00.154922 10580 tablet_bootstrap.cc:492] T 434c464e11b64a00ae9d3d3097ca345d P 9416a559822d4196a77a162a8bb09b99: No bootstrap required, opened a new log
I20240502 08:04:00.155438 10580 ts_tablet_manager.cc:1367] T 434c464e11b64a00ae9d3d3097ca345d P 9416a559822d4196a77a162a8bb09b99: Time spent bootstrapping tablet: real 0.023s user 0.005s sys 0.005s
I20240502 08:04:00.156477 10581 tablet_bootstrap.cc:654] T 385fa1c24869499ab21a9a6852f82494 P 71013a2b500f4319bb04ffd933d71a8f: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:00.158028 10580 raft_consensus.cc:348] T 434c464e11b64a00ae9d3d3097ca345d P 9416a559822d4196a77a162a8bb09b99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416a559822d4196a77a162a8bb09b99" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 34737 } } peers { permanent_uuid: "71013a2b500f4319bb04ffd933d71a8f" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 42107 } } peers { permanent_uuid: "9b42bd3f172344d59014088f13911462" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38051 } }
I20240502 08:04:00.158634 10580 raft_consensus.cc:374] T 434c464e11b64a00ae9d3d3097ca345d P 9416a559822d4196a77a162a8bb09b99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:00.158881 10580 raft_consensus.cc:729] T 434c464e11b64a00ae9d3d3097ca345d P 9416a559822d4196a77a162a8bb09b99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9416a559822d4196a77a162a8bb09b99, State: Initialized, Role: FOLLOWER
I20240502 08:04:00.160722 10582 tablet_bootstrap.cc:492] T 385fa1c24869499ab21a9a6852f82494 P 9b42bd3f172344d59014088f13911462: No bootstrap required, opened a new log
I20240502 08:04:00.161151 10582 ts_tablet_manager.cc:1367] T 385fa1c24869499ab21a9a6852f82494 P 9b42bd3f172344d59014088f13911462: Time spent bootstrapping tablet: real 0.026s user 0.007s sys 0.003s
I20240502 08:04:00.163406 10582 raft_consensus.cc:348] T 385fa1c24869499ab21a9a6852f82494 P 9b42bd3f172344d59014088f13911462 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416a559822d4196a77a162a8bb09b99" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 34737 } } peers { permanent_uuid: "71013a2b500f4319bb04ffd933d71a8f" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 42107 } } peers { permanent_uuid: "9b42bd3f172344d59014088f13911462" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38051 } }
I20240502 08:04:00.163949 10582 raft_consensus.cc:374] T 385fa1c24869499ab21a9a6852f82494 P 9b42bd3f172344d59014088f13911462 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:00.164178 10582 raft_consensus.cc:729] T 385fa1c24869499ab21a9a6852f82494 P 9b42bd3f172344d59014088f13911462 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b42bd3f172344d59014088f13911462, State: Initialized, Role: FOLLOWER
I20240502 08:04:00.164633 10581 tablet_bootstrap.cc:492] T 385fa1c24869499ab21a9a6852f82494 P 71013a2b500f4319bb04ffd933d71a8f: No bootstrap required, opened a new log
I20240502 08:04:00.165052 10581 ts_tablet_manager.cc:1367] T 385fa1c24869499ab21a9a6852f82494 P 71013a2b500f4319bb04ffd933d71a8f: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.004s
I20240502 08:04:00.164736 10582 consensus_queue.cc:260] T 385fa1c24869499ab21a9a6852f82494 P 9b42bd3f172344d59014088f13911462 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416a559822d4196a77a162a8bb09b99" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 34737 } } peers { permanent_uuid: "71013a2b500f4319bb04ffd933d71a8f" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 42107 } } peers { permanent_uuid: "9b42bd3f172344d59014088f13911462" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38051 } }
I20240502 08:04:00.166698 10582 ts_tablet_manager.cc:1398] T 385fa1c24869499ab21a9a6852f82494 P 9b42bd3f172344d59014088f13911462: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20240502 08:04:00.166393 10580 consensus_queue.cc:260] T 434c464e11b64a00ae9d3d3097ca345d P 9416a559822d4196a77a162a8bb09b99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416a559822d4196a77a162a8bb09b99" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 34737 } } peers { permanent_uuid: "71013a2b500f4319bb04ffd933d71a8f" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 42107 } } peers { permanent_uuid: "9b42bd3f172344d59014088f13911462" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38051 } }
I20240502 08:04:00.167795 10582 tablet_bootstrap.cc:492] T b69b32e4d5b04f00956ef81511e7c9eb P 9b42bd3f172344d59014088f13911462: Bootstrap starting.
I20240502 08:04:00.167629 10581 raft_consensus.cc:348] T 385fa1c24869499ab21a9a6852f82494 P 71013a2b500f4319bb04ffd933d71a8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416a559822d4196a77a162a8bb09b99" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 34737 } } peers { permanent_uuid: "71013a2b500f4319bb04ffd933d71a8f" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 42107 } } peers { permanent_uuid: "9b42bd3f172344d59014088f13911462" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38051 } }
I20240502 08:04:00.168455 10581 raft_consensus.cc:374] T 385fa1c24869499ab21a9a6852f82494 P 71013a2b500f4319bb04ffd933d71a8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:00.168812 10580 ts_tablet_manager.cc:1398] T 434c464e11b64a00ae9d3d3097ca345d P 9416a559822d4196a77a162a8bb09b99: Time spent starting tablet: real 0.013s user 0.000s sys 0.006s
I20240502 08:04:00.169965 10580 tablet_bootstrap.cc:492] T b69b32e4d5b04f00956ef81511e7c9eb P 9416a559822d4196a77a162a8bb09b99: Bootstrap starting.
I20240502 08:04:00.168812 10581 raft_consensus.cc:729] T 385fa1c24869499ab21a9a6852f82494 P 71013a2b500f4319bb04ffd933d71a8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71013a2b500f4319bb04ffd933d71a8f, State: Initialized, Role: FOLLOWER
I20240502 08:04:00.172526 10581 consensus_queue.cc:260] T 385fa1c24869499ab21a9a6852f82494 P 71013a2b500f4319bb04ffd933d71a8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416a559822d4196a77a162a8bb09b99" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 34737 } } peers { permanent_uuid: "71013a2b500f4319bb04ffd933d71a8f" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 42107 } } peers { permanent_uuid: "9b42bd3f172344d59014088f13911462" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38051 } }
I20240502 08:04:00.174728 10582 tablet_bootstrap.cc:654] T b69b32e4d5b04f00956ef81511e7c9eb P 9b42bd3f172344d59014088f13911462: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:00.174856 10581 ts_tablet_manager.cc:1398] T 385fa1c24869499ab21a9a6852f82494 P 71013a2b500f4319bb04ffd933d71a8f: Time spent starting tablet: real 0.010s user 0.006s sys 0.001s
I20240502 08:04:00.175963 10581 tablet_bootstrap.cc:492] T 434c464e11b64a00ae9d3d3097ca345d P 71013a2b500f4319bb04ffd933d71a8f: Bootstrap starting.
I20240502 08:04:00.180294 10580 tablet_bootstrap.cc:654] T b69b32e4d5b04f00956ef81511e7c9eb P 9416a559822d4196a77a162a8bb09b99: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:00.180859 10581 tablet_bootstrap.cc:654] T 434c464e11b64a00ae9d3d3097ca345d P 71013a2b500f4319bb04ffd933d71a8f: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:00.187319 10582 tablet_bootstrap.cc:492] T b69b32e4d5b04f00956ef81511e7c9eb P 9b42bd3f172344d59014088f13911462: No bootstrap required, opened a new log
I20240502 08:04:00.187896 10582 ts_tablet_manager.cc:1367] T b69b32e4d5b04f00956ef81511e7c9eb P 9b42bd3f172344d59014088f13911462: Time spent bootstrapping tablet: real 0.020s user 0.012s sys 0.001s
I20240502 08:04:00.188750 10580 tablet_bootstrap.cc:492] T b69b32e4d5b04f00956ef81511e7c9eb P 9416a559822d4196a77a162a8bb09b99: No bootstrap required, opened a new log
I20240502 08:04:00.189177 10581 tablet_bootstrap.cc:492] T 434c464e11b64a00ae9d3d3097ca345d P 71013a2b500f4319bb04ffd933d71a8f: No bootstrap required, opened a new log
I20240502 08:04:00.189285 10580 ts_tablet_manager.cc:1367] T b69b32e4d5b04f00956ef81511e7c9eb P 9416a559822d4196a77a162a8bb09b99: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.005s
I20240502 08:04:00.189699 10581 ts_tablet_manager.cc:1367] T 434c464e11b64a00ae9d3d3097ca345d P 71013a2b500f4319bb04ffd933d71a8f: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.004s
I20240502 08:04:00.190856 10582 raft_consensus.cc:348] T b69b32e4d5b04f00956ef81511e7c9eb P 9b42bd3f172344d59014088f13911462 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416a559822d4196a77a162a8bb09b99" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 34737 } } peers { permanent_uuid: "71013a2b500f4319bb04ffd933d71a8f" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 42107 } } peers { permanent_uuid: "9b42bd3f172344d59014088f13911462" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38051 } }
I20240502 08:04:00.191756 10582 raft_consensus.cc:374] T b69b32e4d5b04f00956ef81511e7c9eb P 9b42bd3f172344d59014088f13911462 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:00.192126 10582 raft_consensus.cc:729] T b69b32e4d5b04f00956ef81511e7c9eb P 9b42bd3f172344d59014088f13911462 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b42bd3f172344d59014088f13911462, State: Initialized, Role: FOLLOWER
I20240502 08:04:00.192075 10580 raft_consensus.cc:348] T b69b32e4d5b04f00956ef81511e7c9eb P 9416a559822d4196a77a162a8bb09b99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416a559822d4196a77a162a8bb09b99" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 34737 } } peers { permanent_uuid: "71013a2b500f4319bb04ffd933d71a8f" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 42107 } } peers { permanent_uuid: "9b42bd3f172344d59014088f13911462" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38051 } }
I20240502 08:04:00.192332 10581 raft_consensus.cc:348] T 434c464e11b64a00ae9d3d3097ca345d P 71013a2b500f4319bb04ffd933d71a8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416a559822d4196a77a162a8bb09b99" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 34737 } } peers { permanent_uuid: "71013a2b500f4319bb04ffd933d71a8f" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 42107 } } peers { permanent_uuid: "9b42bd3f172344d59014088f13911462" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38051 } }
I20240502 08:04:00.193063 10580 raft_consensus.cc:374] T b69b32e4d5b04f00956ef81511e7c9eb P 9416a559822d4196a77a162a8bb09b99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:00.193163 10581 raft_consensus.cc:374] T 434c464e11b64a00ae9d3d3097ca345d P 71013a2b500f4319bb04ffd933d71a8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:00.193481 10580 raft_consensus.cc:729] T b69b32e4d5b04f00956ef81511e7c9eb P 9416a559822d4196a77a162a8bb09b99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9416a559822d4196a77a162a8bb09b99, State: Initialized, Role: FOLLOWER
I20240502 08:04:00.193588 10581 raft_consensus.cc:729] T 434c464e11b64a00ae9d3d3097ca345d P 71013a2b500f4319bb04ffd933d71a8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71013a2b500f4319bb04ffd933d71a8f, State: Initialized, Role: FOLLOWER
I20240502 08:04:00.193298 10582 consensus_queue.cc:260] T b69b32e4d5b04f00956ef81511e7c9eb P 9b42bd3f172344d59014088f13911462 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416a559822d4196a77a162a8bb09b99" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 34737 } } peers { permanent_uuid: "71013a2b500f4319bb04ffd933d71a8f" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 42107 } } peers { permanent_uuid: "9b42bd3f172344d59014088f13911462" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38051 } }
I20240502 08:04:00.194442 10580 consensus_queue.cc:260] T b69b32e4d5b04f00956ef81511e7c9eb P 9416a559822d4196a77a162a8bb09b99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416a559822d4196a77a162a8bb09b99" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 34737 } } peers { permanent_uuid: "71013a2b500f4319bb04ffd933d71a8f" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 42107 } } peers { permanent_uuid: "9b42bd3f172344d59014088f13911462" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38051 } }
I20240502 08:04:00.194438 10581 consensus_queue.cc:260] T 434c464e11b64a00ae9d3d3097ca345d P 71013a2b500f4319bb04ffd933d71a8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416a559822d4196a77a162a8bb09b99" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 34737 } } peers { permanent_uuid: "71013a2b500f4319bb04ffd933d71a8f" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 42107 } } peers { permanent_uuid: "9b42bd3f172344d59014088f13911462" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38051 } }
I20240502 08:04:00.195995 10582 ts_tablet_manager.cc:1398] T b69b32e4d5b04f00956ef81511e7c9eb P 9b42bd3f172344d59014088f13911462: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20240502 08:04:00.198493 10580 ts_tablet_manager.cc:1398] T b69b32e4d5b04f00956ef81511e7c9eb P 9416a559822d4196a77a162a8bb09b99: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20240502 08:04:00.198514 10581 ts_tablet_manager.cc:1398] T 434c464e11b64a00ae9d3d3097ca345d P 71013a2b500f4319bb04ffd933d71a8f: Time spent starting tablet: real 0.008s user 0.003s sys 0.003s
I20240502 08:04:00.199901 10588 raft_consensus.cc:482] T b69b32e4d5b04f00956ef81511e7c9eb P 71013a2b500f4319bb04ffd933d71a8f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:04:00.200379 10588 raft_consensus.cc:504] T b69b32e4d5b04f00956ef81511e7c9eb P 71013a2b500f4319bb04ffd933d71a8f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416a559822d4196a77a162a8bb09b99" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 34737 } } peers { permanent_uuid: "71013a2b500f4319bb04ffd933d71a8f" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 42107 } } peers { permanent_uuid: "9b42bd3f172344d59014088f13911462" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38051 } }
I20240502 08:04:00.202845 10588 leader_election.cc:288] T b69b32e4d5b04f00956ef81511e7c9eb P 71013a2b500f4319bb04ffd933d71a8f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9416a559822d4196a77a162a8bb09b99 (127.6.204.193:34737), 9b42bd3f172344d59014088f13911462 (127.6.204.195:38051)
I20240502 08:04:00.215796 10386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b69b32e4d5b04f00956ef81511e7c9eb" candidate_uuid: "71013a2b500f4319bb04ffd933d71a8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9416a559822d4196a77a162a8bb09b99" is_pre_election: true
I20240502 08:04:00.216722 10386 raft_consensus.cc:2454] T b69b32e4d5b04f00956ef81511e7c9eb P 9416a559822d4196a77a162a8bb09b99 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71013a2b500f4319bb04ffd933d71a8f in term 0.
I20240502 08:04:00.217952 10426 leader_election.cc:302] T b69b32e4d5b04f00956ef81511e7c9eb P 71013a2b500f4319bb04ffd933d71a8f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71013a2b500f4319bb04ffd933d71a8f, 9416a559822d4196a77a162a8bb09b99; no voters:
I20240502 08:04:00.218842 10588 raft_consensus.cc:2781] T b69b32e4d5b04f00956ef81511e7c9eb P 71013a2b500f4319bb04ffd933d71a8f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:04:00.219141 10588 raft_consensus.cc:482] T b69b32e4d5b04f00956ef81511e7c9eb P 71013a2b500f4319bb04ffd933d71a8f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:04:00.219390 10588 raft_consensus.cc:3037] T b69b32e4d5b04f00956ef81511e7c9eb P 71013a2b500f4319bb04ffd933d71a8f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:00.221892 10535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b69b32e4d5b04f00956ef81511e7c9eb" candidate_uuid: "71013a2b500f4319bb04ffd933d71a8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b42bd3f172344d59014088f13911462" is_pre_election: true
I20240502 08:04:00.222501 10535 raft_consensus.cc:2454] T b69b32e4d5b04f00956ef81511e7c9eb P 9b42bd3f172344d59014088f13911462 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71013a2b500f4319bb04ffd933d71a8f in term 0.
I20240502 08:04:00.224354 10597 raft_consensus.cc:482] T 385fa1c24869499ab21a9a6852f82494 P 71013a2b500f4319bb04ffd933d71a8f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:04:00.224825 10597 raft_consensus.cc:504] T 385fa1c24869499ab21a9a6852f82494 P 71013a2b500f4319bb04ffd933d71a8f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416a559822d4196a77a162a8bb09b99" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 34737 } } peers { permanent_uuid: "71013a2b500f4319bb04ffd933d71a8f" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 42107 } } peers { permanent_uuid: "9b42bd3f172344d59014088f13911462" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38051 } }
I20240502 08:04:00.225495 10588 raft_consensus.cc:504] T b69b32e4d5b04f00956ef81511e7c9eb P 71013a2b500f4319bb04ffd933d71a8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416a559822d4196a77a162a8bb09b99" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 34737 } } peers { permanent_uuid: "71013a2b500f4319bb04ffd933d71a8f" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 42107 } } peers { permanent_uuid: "9b42bd3f172344d59014088f13911462" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38051 } }
I20240502 08:04:00.227418 10588 leader_election.cc:288] T b69b32e4d5b04f00956ef81511e7c9eb P 71013a2b500f4319bb04ffd933d71a8f [CANDIDATE]: Term 1 election: Requested vote from peers 9416a559822d4196a77a162a8bb09b99 (127.6.204.193:34737), 9b42bd3f172344d59014088f13911462 (127.6.204.195:38051)
I20240502 08:04:00.227958 10535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "385fa1c24869499ab21a9a6852f82494" candidate_uuid: "71013a2b500f4319bb04ffd933d71a8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b42bd3f172344d59014088f13911462" is_pre_election: true
I20240502 08:04:00.228015 10386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "385fa1c24869499ab21a9a6852f82494" candidate_uuid: "71013a2b500f4319bb04ffd933d71a8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9416a559822d4196a77a162a8bb09b99" is_pre_election: true
I20240502 08:04:00.228507 10535 raft_consensus.cc:2454] T 385fa1c24869499ab21a9a6852f82494 P 9b42bd3f172344d59014088f13911462 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71013a2b500f4319bb04ffd933d71a8f in term 0.
I20240502 08:04:00.228569 10386 raft_consensus.cc:2454] T 385fa1c24869499ab21a9a6852f82494 P 9416a559822d4196a77a162a8bb09b99 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71013a2b500f4319bb04ffd933d71a8f in term 0.
I20240502 08:04:00.229449 10426 leader_election.cc:302] T 385fa1c24869499ab21a9a6852f82494 P 71013a2b500f4319bb04ffd933d71a8f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71013a2b500f4319bb04ffd933d71a8f, 9416a559822d4196a77a162a8bb09b99; no voters:
I20240502 08:04:00.229600 10385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b69b32e4d5b04f00956ef81511e7c9eb" candidate_uuid: "71013a2b500f4319bb04ffd933d71a8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9416a559822d4196a77a162a8bb09b99"
I20240502 08:04:00.230110 10588 raft_consensus.cc:2781] T 385fa1c24869499ab21a9a6852f82494 P 71013a2b500f4319bb04ffd933d71a8f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:04:00.230177 10385 raft_consensus.cc:3037] T b69b32e4d5b04f00956ef81511e7c9eb P 9416a559822d4196a77a162a8bb09b99 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:00.230657 10588 raft_consensus.cc:482] T 385fa1c24869499ab21a9a6852f82494 P 71013a2b500f4319bb04ffd933d71a8f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:04:00.231006 10588 raft_consensus.cc:3037] T 385fa1c24869499ab21a9a6852f82494 P 71013a2b500f4319bb04ffd933d71a8f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:00.231106 10535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b69b32e4d5b04f00956ef81511e7c9eb" candidate_uuid: "71013a2b500f4319bb04ffd933d71a8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b42bd3f172344d59014088f13911462"
I20240502 08:04:00.231551 10535 raft_consensus.cc:3037] T b69b32e4d5b04f00956ef81511e7c9eb P 9b42bd3f172344d59014088f13911462 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:00.234268 10597 leader_election.cc:288] T 385fa1c24869499ab21a9a6852f82494 P 71013a2b500f4319bb04ffd933d71a8f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9416a559822d4196a77a162a8bb09b99 (127.6.204.193:34737), 9b42bd3f172344d59014088f13911462 (127.6.204.195:38051)
I20240502 08:04:00.236817 10535 raft_consensus.cc:2454] T b69b32e4d5b04f00956ef81511e7c9eb P 9b42bd3f172344d59014088f13911462 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71013a2b500f4319bb04ffd933d71a8f in term 1.
I20240502 08:04:00.237496 10385 raft_consensus.cc:2454] T b69b32e4d5b04f00956ef81511e7c9eb P 9416a559822d4196a77a162a8bb09b99 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71013a2b500f4319bb04ffd933d71a8f in term 1.
I20240502 08:04:00.237452 10588 raft_consensus.cc:504] T 385fa1c24869499ab21a9a6852f82494 P 71013a2b500f4319bb04ffd933d71a8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416a559822d4196a77a162a8bb09b99" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 34737 } } peers { permanent_uuid: "71013a2b500f4319bb04ffd933d71a8f" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 42107 } } peers { permanent_uuid: "9b42bd3f172344d59014088f13911462" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38051 } }
I20240502 08:04:00.237890 10427 leader_election.cc:302] T b69b32e4d5b04f00956ef81511e7c9eb P 71013a2b500f4319bb04ffd933d71a8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71013a2b500f4319bb04ffd933d71a8f, 9b42bd3f172344d59014088f13911462; no voters:
I20240502 08:04:00.238767 10597 raft_consensus.cc:2781] T b69b32e4d5b04f00956ef81511e7c9eb P 71013a2b500f4319bb04ffd933d71a8f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:04:00.239645 10588 leader_election.cc:288] T 385fa1c24869499ab21a9a6852f82494 P 71013a2b500f4319bb04ffd933d71a8f [CANDIDATE]: Term 1 election: Requested vote from peers 9416a559822d4196a77a162a8bb09b99 (127.6.204.193:34737), 9b42bd3f172344d59014088f13911462 (127.6.204.195:38051)
I20240502 08:04:00.240295 10385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "385fa1c24869499ab21a9a6852f82494" candidate_uuid: "71013a2b500f4319bb04ffd933d71a8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9416a559822d4196a77a162a8bb09b99"
I20240502 08:04:00.241550 10385 raft_consensus.cc:3037] T 385fa1c24869499ab21a9a6852f82494 P 9416a559822d4196a77a162a8bb09b99 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:00.241983 10535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "385fa1c24869499ab21a9a6852f82494" candidate_uuid: "71013a2b500f4319bb04ffd933d71a8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b42bd3f172344d59014088f13911462"
I20240502 08:04:00.242576 10535 raft_consensus.cc:3037] T 385fa1c24869499ab21a9a6852f82494 P 9b42bd3f172344d59014088f13911462 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:00.248004 10385 raft_consensus.cc:2454] T 385fa1c24869499ab21a9a6852f82494 P 9416a559822d4196a77a162a8bb09b99 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71013a2b500f4319bb04ffd933d71a8f in term 1.
I20240502 08:04:00.248867 10535 raft_consensus.cc:2454] T 385fa1c24869499ab21a9a6852f82494 P 9b42bd3f172344d59014088f13911462 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71013a2b500f4319bb04ffd933d71a8f in term 1.
I20240502 08:04:00.249254 10426 leader_election.cc:302] T 385fa1c24869499ab21a9a6852f82494 P 71013a2b500f4319bb04ffd933d71a8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71013a2b500f4319bb04ffd933d71a8f, 9416a559822d4196a77a162a8bb09b99; no voters:
I20240502 08:04:00.250216 10588 raft_consensus.cc:2781] T 385fa1c24869499ab21a9a6852f82494 P 71013a2b500f4319bb04ffd933d71a8f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:04:00.250409 10597 raft_consensus.cc:686] T b69b32e4d5b04f00956ef81511e7c9eb P 71013a2b500f4319bb04ffd933d71a8f [term 1 LEADER]: Becoming Leader. State: Replica: 71013a2b500f4319bb04ffd933d71a8f, State: Running, Role: LEADER
I20240502 08:04:00.250758 10588 raft_consensus.cc:686] T 385fa1c24869499ab21a9a6852f82494 P 71013a2b500f4319bb04ffd933d71a8f [term 1 LEADER]: Becoming Leader. State: Replica: 71013a2b500f4319bb04ffd933d71a8f, State: Running, Role: LEADER
I20240502 08:04:00.251356 10597 consensus_queue.cc:237] T b69b32e4d5b04f00956ef81511e7c9eb P 71013a2b500f4319bb04ffd933d71a8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416a559822d4196a77a162a8bb09b99" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 34737 } } peers { permanent_uuid: "71013a2b500f4319bb04ffd933d71a8f" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 42107 } } peers { permanent_uuid: "9b42bd3f172344d59014088f13911462" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38051 } }
I20240502 08:04:00.251518 10588 consensus_queue.cc:237] T 385fa1c24869499ab21a9a6852f82494 P 71013a2b500f4319bb04ffd933d71a8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416a559822d4196a77a162a8bb09b99" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 34737 } } peers { permanent_uuid: "71013a2b500f4319bb04ffd933d71a8f" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 42107 } } peers { permanent_uuid: "9b42bd3f172344d59014088f13911462" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38051 } }
I20240502 08:04:00.258793 10281 catalog_manager.cc:5484] T b69b32e4d5b04f00956ef81511e7c9eb P 71013a2b500f4319bb04ffd933d71a8f reported cstate change: term changed from 0 to 1, leader changed from <none> to 71013a2b500f4319bb04ffd933d71a8f (127.6.204.194). New cstate: current_term: 1 leader_uuid: "71013a2b500f4319bb04ffd933d71a8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416a559822d4196a77a162a8bb09b99" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 34737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71013a2b500f4319bb04ffd933d71a8f" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 42107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b42bd3f172344d59014088f13911462" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38051 } health_report { overall_health: UNKNOWN } } }
I20240502 08:04:00.269793 10281 catalog_manager.cc:5484] T 385fa1c24869499ab21a9a6852f82494 P 71013a2b500f4319bb04ffd933d71a8f reported cstate change: term changed from 0 to 1, leader changed from <none> to 71013a2b500f4319bb04ffd933d71a8f (127.6.204.194). New cstate: current_term: 1 leader_uuid: "71013a2b500f4319bb04ffd933d71a8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416a559822d4196a77a162a8bb09b99" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 34737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71013a2b500f4319bb04ffd933d71a8f" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 42107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b42bd3f172344d59014088f13911462" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38051 } health_report { overall_health: UNKNOWN } } }
I20240502 08:04:00.314833 10587 raft_consensus.cc:482] T 434c464e11b64a00ae9d3d3097ca345d P 9416a559822d4196a77a162a8bb09b99 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:04:00.315335 10587 raft_consensus.cc:504] T 434c464e11b64a00ae9d3d3097ca345d P 9416a559822d4196a77a162a8bb09b99 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416a559822d4196a77a162a8bb09b99" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 34737 } } peers { permanent_uuid: "71013a2b500f4319bb04ffd933d71a8f" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 42107 } } peers { permanent_uuid: "9b42bd3f172344d59014088f13911462" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38051 } }
I20240502 08:04:00.317313 10587 leader_election.cc:288] T 434c464e11b64a00ae9d3d3097ca345d P 9416a559822d4196a77a162a8bb09b99 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71013a2b500f4319bb04ffd933d71a8f (127.6.204.194:42107), 9b42bd3f172344d59014088f13911462 (127.6.204.195:38051)
I20240502 08:04:00.327713 10461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "434c464e11b64a00ae9d3d3097ca345d" candidate_uuid: "9416a559822d4196a77a162a8bb09b99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71013a2b500f4319bb04ffd933d71a8f" is_pre_election: true
I20240502 08:04:00.327806 10535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "434c464e11b64a00ae9d3d3097ca345d" candidate_uuid: "9416a559822d4196a77a162a8bb09b99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b42bd3f172344d59014088f13911462" is_pre_election: true
I20240502 08:04:00.328428 10461 raft_consensus.cc:2454] T 434c464e11b64a00ae9d3d3097ca345d P 71013a2b500f4319bb04ffd933d71a8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9416a559822d4196a77a162a8bb09b99 in term 0.
I20240502 08:04:00.328497 10535 raft_consensus.cc:2454] T 434c464e11b64a00ae9d3d3097ca345d P 9b42bd3f172344d59014088f13911462 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9416a559822d4196a77a162a8bb09b99 in term 0.
I20240502 08:04:00.329476 10352 leader_election.cc:302] T 434c464e11b64a00ae9d3d3097ca345d P 9416a559822d4196a77a162a8bb09b99 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9416a559822d4196a77a162a8bb09b99, 9b42bd3f172344d59014088f13911462; no voters:
I20240502 08:04:00.330224 10587 raft_consensus.cc:2781] T 434c464e11b64a00ae9d3d3097ca345d P 9416a559822d4196a77a162a8bb09b99 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:04:00.330580 10587 raft_consensus.cc:482] T 434c464e11b64a00ae9d3d3097ca345d P 9416a559822d4196a77a162a8bb09b99 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:04:00.330865 10587 raft_consensus.cc:3037] T 434c464e11b64a00ae9d3d3097ca345d P 9416a559822d4196a77a162a8bb09b99 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:00.335520 10587 raft_consensus.cc:504] T 434c464e11b64a00ae9d3d3097ca345d P 9416a559822d4196a77a162a8bb09b99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416a559822d4196a77a162a8bb09b99" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 34737 } } peers { permanent_uuid: "71013a2b500f4319bb04ffd933d71a8f" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 42107 } } peers { permanent_uuid: "9b42bd3f172344d59014088f13911462" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38051 } }
I20240502 08:04:00.336967 10587 leader_election.cc:288] T 434c464e11b64a00ae9d3d3097ca345d P 9416a559822d4196a77a162a8bb09b99 [CANDIDATE]: Term 1 election: Requested vote from peers 71013a2b500f4319bb04ffd933d71a8f (127.6.204.194:42107), 9b42bd3f172344d59014088f13911462 (127.6.204.195:38051)
I20240502 08:04:00.337921 10535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "434c464e11b64a00ae9d3d3097ca345d" candidate_uuid: "9416a559822d4196a77a162a8bb09b99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b42bd3f172344d59014088f13911462"
I20240502 08:04:00.337831 10461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "434c464e11b64a00ae9d3d3097ca345d" candidate_uuid: "9416a559822d4196a77a162a8bb09b99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71013a2b500f4319bb04ffd933d71a8f"
I20240502 08:04:00.338567 10535 raft_consensus.cc:3037] T 434c464e11b64a00ae9d3d3097ca345d P 9b42bd3f172344d59014088f13911462 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:00.338639 10461 raft_consensus.cc:3037] T 434c464e11b64a00ae9d3d3097ca345d P 71013a2b500f4319bb04ffd933d71a8f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:00.345412 10461 raft_consensus.cc:2454] T 434c464e11b64a00ae9d3d3097ca345d P 71013a2b500f4319bb04ffd933d71a8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9416a559822d4196a77a162a8bb09b99 in term 1.
I20240502 08:04:00.346978 10350 leader_election.cc:302] T 434c464e11b64a00ae9d3d3097ca345d P 9416a559822d4196a77a162a8bb09b99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71013a2b500f4319bb04ffd933d71a8f, 9416a559822d4196a77a162a8bb09b99; no voters:
I20240502 08:04:00.345412 10535 raft_consensus.cc:2454] T 434c464e11b64a00ae9d3d3097ca345d P 9b42bd3f172344d59014088f13911462 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9416a559822d4196a77a162a8bb09b99 in term 1.
I20240502 08:04:00.353560 10587 raft_consensus.cc:2781] T 434c464e11b64a00ae9d3d3097ca345d P 9416a559822d4196a77a162a8bb09b99 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:04:00.355144 10587 raft_consensus.cc:686] T 434c464e11b64a00ae9d3d3097ca345d P 9416a559822d4196a77a162a8bb09b99 [term 1 LEADER]: Becoming Leader. State: Replica: 9416a559822d4196a77a162a8bb09b99, State: Running, Role: LEADER
I20240502 08:04:00.356042 10587 consensus_queue.cc:237] T 434c464e11b64a00ae9d3d3097ca345d P 9416a559822d4196a77a162a8bb09b99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416a559822d4196a77a162a8bb09b99" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 34737 } } peers { permanent_uuid: "71013a2b500f4319bb04ffd933d71a8f" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 42107 } } peers { permanent_uuid: "9b42bd3f172344d59014088f13911462" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38051 } }
I20240502 08:04:00.363581 10281 catalog_manager.cc:5484] T 434c464e11b64a00ae9d3d3097ca345d P 9416a559822d4196a77a162a8bb09b99 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9416a559822d4196a77a162a8bb09b99 (127.6.204.193). New cstate: current_term: 1 leader_uuid: "9416a559822d4196a77a162a8bb09b99" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416a559822d4196a77a162a8bb09b99" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 34737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71013a2b500f4319bb04ffd933d71a8f" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 42107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b42bd3f172344d59014088f13911462" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38051 } health_report { overall_health: UNKNOWN } } }
I20240502 08:04:00.431239 6963 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:04:00.437587 10605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:04:00.439014 10606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:04:00.440598 6963 server_base.cc:1034] running on GCE node
W20240502 08:04:00.440595 10608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:04:00.442339 6963 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:04:00.442584 6963 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:04:00.442778 6963 hybrid_clock.cc:648] HybridClock initialized: now 1714637040442760 us; error 0 us; skew 500 ppm
I20240502 08:04:00.443336 6963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:04:00.446008 6963 webserver.cc:460] Webserver started at http://127.6.204.196:32939/ using document root <none> and password file <none>
I20240502 08:04:00.446542 6963 fs_manager.cc:359] Metadata directory not provided
I20240502 08:04:00.446713 6963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:04:00.447001 6963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:04:00.448135 6963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714636933625983-6963-0/minicluster-data/ts-3-root/instance:
uuid: "d202fa5d5d3a468483a3402915d7e8ac"
format_stamp: "Formatted at 2024-05-02 08:04:00 on dist-test-slave-32xs"
I20240502 08:04:00.452993 6963 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20240502 08:04:00.456511 10613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:04:00.457310 6963 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 08:04:00.457624 6963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714636933625983-6963-0/minicluster-data/ts-3-root
uuid: "d202fa5d5d3a468483a3402915d7e8ac"
format_stamp: "Formatted at 2024-05-02 08:04:00 on dist-test-slave-32xs"
I20240502 08:04:00.457916 6963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714636933625983-6963-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714636933625983-6963-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714636933625983-6963-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:04:00.484303 6963 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:04:00.485517 6963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:04:00.487520 6963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:04:00.489802 6963 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:04:00.490006 6963 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:04:00.490239 6963 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:04:00.490388 6963 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:04:00.502313 10597 consensus_queue.cc:1035] T b69b32e4d5b04f00956ef81511e7c9eb P 71013a2b500f4319bb04ffd933d71a8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9416a559822d4196a77a162a8bb09b99" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 34737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20240502 08:04:00.510855 10598 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20240502 08:04:00.511245 10598 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20240502 08:04:00.536437 10588 consensus_queue.cc:1035] T 385fa1c24869499ab21a9a6852f82494 P 71013a2b500f4319bb04ffd933d71a8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b42bd3f172344d59014088f13911462" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 08:04:00.543200 6963 rpc_server.cc:307] RPC server started. Bound to: 127.6.204.196:41623
I20240502 08:04:00.543311 10678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.204.196:41623 every 8 connection(s)
I20240502 08:04:00.554996 10588 consensus_queue.cc:1035] T 385fa1c24869499ab21a9a6852f82494 P 71013a2b500f4319bb04ffd933d71a8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9416a559822d4196a77a162a8bb09b99" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 34737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 08:04:00.567678 10679 heartbeater.cc:347] Connected to a master server at 127.6.204.254:46313
I20240502 08:04:00.568212 10679 heartbeater.cc:464] Registering TS with master...
I20240502 08:04:00.569530 10679 heartbeater.cc:511] Master 127.6.204.254:46313 requested a full tablet report, sending...
I20240502 08:04:00.572569 10284 ts_manager.cc:194] Registered new tserver with Master: d202fa5d5d3a468483a3402915d7e8ac (127.6.204.196:41623)
I20240502 08:04:00.574553 10284 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47380
I20240502 08:04:00.577808 10684 consensus_queue.cc:1035] T b69b32e4d5b04f00956ef81511e7c9eb P 71013a2b500f4319bb04ffd933d71a8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b42bd3f172344d59014088f13911462" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:04:00.584162 10337 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:04:00.604197 10461 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "385fa1c24869499ab21a9a6852f82494"
dest_uuid: "71013a2b500f4319bb04ffd933d71a8f"
mode: GRACEFUL
new_leader_uuid: "9b42bd3f172344d59014088f13911462"
from {username='slave'} at 127.0.0.1:46536
I20240502 08:04:00.605011 10461 raft_consensus.cc:595] T 385fa1c24869499ab21a9a6852f82494 P 71013a2b500f4319bb04ffd933d71a8f [term 1 LEADER]: Received request to transfer leadership to TS 9b42bd3f172344d59014088f13911462
I20240502 08:04:00.606292 10337 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240502 08:04:00.606693 10337 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
W20240502 08:04:00.611374 10332 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Leader has not yet committed an operation in its own term
W20240502 08:04:00.615118 10332 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 434c464e11b64a00ae9d3d3097ca345d, TS 71013a2b500f4319bb04ffd933d71a8f -> TS d202fa5d5d3a468483a3402915d7e8ac move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416a559822d4196a77a162a8bb09b99" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 34737 } } peers { permanent_uuid: "71013a2b500f4319bb04ffd933d71a8f" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 42107 } } peers { permanent_uuid: "9b42bd3f172344d59014088f13911462" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38051 } }
W20240502 08:04:00.615402 10332 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 434c464e11b64a00ae9d3d3097ca345d, TS 71013a2b500f4319bb04ffd933d71a8f -> TS d202fa5d5d3a468483a3402915d7e8ac move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416a559822d4196a77a162a8bb09b99" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 34737 } } peers { permanent_uuid: "71013a2b500f4319bb04ffd933d71a8f" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 42107 } } peers { permanent_uuid: "9b42bd3f172344d59014088f13911462" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38051 } }
W20240502 08:04:00.619004 10332 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet b69b32e4d5b04f00956ef81511e7c9eb, TS 9b42bd3f172344d59014088f13911462 -> TS d202fa5d5d3a468483a3402915d7e8ac move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416a559822d4196a77a162a8bb09b99" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 34737 } } peers { permanent_uuid: "71013a2b500f4319bb04ffd933d71a8f" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 42107 } } peers { permanent_uuid: "9b42bd3f172344d59014088f13911462" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38051 } }
W20240502 08:04:00.619257 10332 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet b69b32e4d5b04f00956ef81511e7c9eb, TS 9b42bd3f172344d59014088f13911462 -> TS d202fa5d5d3a468483a3402915d7e8ac move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416a559822d4196a77a162a8bb09b99" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 34737 } } peers { permanent_uuid: "71013a2b500f4319bb04ffd933d71a8f" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 42107 } } peers { permanent_uuid: "9b42bd3f172344d59014088f13911462" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38051 } }
I20240502 08:04:00.633431 10587 consensus_queue.cc:1035] T 434c464e11b64a00ae9d3d3097ca345d P 9416a559822d4196a77a162a8bb09b99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71013a2b500f4319bb04ffd933d71a8f" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 42107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:04:00.650264 10587 consensus_queue.cc:1035] T 434c464e11b64a00ae9d3d3097ca345d P 9416a559822d4196a77a162a8bb09b99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b42bd3f172344d59014088f13911462" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 08:04:00.924247 10684 raft_consensus.cc:979] T 385fa1c24869499ab21a9a6852f82494 P 71013a2b500f4319bb04ffd933d71a8f: : Instructing follower 9b42bd3f172344d59014088f13911462 to start an election
I20240502 08:04:00.924885 10597 raft_consensus.cc:1067] T 385fa1c24869499ab21a9a6852f82494 P 71013a2b500f4319bb04ffd933d71a8f [term 1 LEADER]: Signalling peer 9b42bd3f172344d59014088f13911462 to start an election
I20240502 08:04:00.926661 10535 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "385fa1c24869499ab21a9a6852f82494"
dest_uuid: "9b42bd3f172344d59014088f13911462"
from {username='slave'} at 127.0.0.1:42796
I20240502 08:04:00.927160 10535 raft_consensus.cc:482] T 385fa1c24869499ab21a9a6852f82494 P 9b42bd3f172344d59014088f13911462 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:04:00.927417 10535 raft_consensus.cc:3037] T 385fa1c24869499ab21a9a6852f82494 P 9b42bd3f172344d59014088f13911462 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:04:00.932898 10535 raft_consensus.cc:504] T 385fa1c24869499ab21a9a6852f82494 P 9b42bd3f172344d59014088f13911462 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416a559822d4196a77a162a8bb09b99" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 34737 } } peers { permanent_uuid: "71013a2b500f4319bb04ffd933d71a8f" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 42107 } } peers { permanent_uuid: "9b42bd3f172344d59014088f13911462" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38051 } }
I20240502 08:04:00.934901 10535 leader_election.cc:288] T 385fa1c24869499ab21a9a6852f82494 P 9b42bd3f172344d59014088f13911462 [CANDIDATE]: Term 2 election: Requested vote from peers 9416a559822d4196a77a162a8bb09b99 (127.6.204.193:34737), 71013a2b500f4319bb04ffd933d71a8f (127.6.204.194:42107)
I20240502 08:04:00.945523 10461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "385fa1c24869499ab21a9a6852f82494" candidate_uuid: "9b42bd3f172344d59014088f13911462" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "71013a2b500f4319bb04ffd933d71a8f"
I20240502 08:04:00.945524 10385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "385fa1c24869499ab21a9a6852f82494" candidate_uuid: "9b42bd3f172344d59014088f13911462" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9416a559822d4196a77a162a8bb09b99"
I20240502 08:04:00.946193 10461 raft_consensus.cc:3032] T 385fa1c24869499ab21a9a6852f82494 P 71013a2b500f4319bb04ffd933d71a8f [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:04:00.946228 10385 raft_consensus.cc:3037] T 385fa1c24869499ab21a9a6852f82494 P 9416a559822d4196a77a162a8bb09b99 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:04:00.947522 10461 raft_consensus.cc:729] T 385fa1c24869499ab21a9a6852f82494 P 71013a2b500f4319bb04ffd933d71a8f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 71013a2b500f4319bb04ffd933d71a8f, State: Running, Role: LEADER
I20240502 08:04:00.948362 10461 consensus_queue.cc:260] T 385fa1c24869499ab21a9a6852f82494 P 71013a2b500f4319bb04ffd933d71a8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416a559822d4196a77a162a8bb09b99" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 34737 } } peers { permanent_uuid: "71013a2b500f4319bb04ffd933d71a8f" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 42107 } } peers { permanent_uuid: "9b42bd3f172344d59014088f13911462" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38051 } }
I20240502 08:04:00.949697 10461 raft_consensus.cc:3037] T 385fa1c24869499ab21a9a6852f82494 P 71013a2b500f4319bb04ffd933d71a8f [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:04:00.953505 10385 raft_consensus.cc:2454] T 385fa1c24869499ab21a9a6852f82494 P 9416a559822d4196a77a162a8bb09b99 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b42bd3f172344d59014088f13911462 in term 2.
I20240502 08:04:00.954634 10500 leader_election.cc:302] T 385fa1c24869499ab21a9a6852f82494 P 9b42bd3f172344d59014088f13911462 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9416a559822d4196a77a162a8bb09b99, 9b42bd3f172344d59014088f13911462; no voters:
I20240502 08:04:00.955332 10585 raft_consensus.cc:2781] T 385fa1c24869499ab21a9a6852f82494 P 9b42bd3f172344d59014088f13911462 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:04:00.955847 10461 raft_consensus.cc:2454] T 385fa1c24869499ab21a9a6852f82494 P 71013a2b500f4319bb04ffd933d71a8f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b42bd3f172344d59014088f13911462 in term 2.
I20240502 08:04:00.956553 10585 raft_consensus.cc:686] T 385fa1c24869499ab21a9a6852f82494 P 9b42bd3f172344d59014088f13911462 [term 2 LEADER]: Becoming Leader. State: Replica: 9b42bd3f172344d59014088f13911462, State: Running, Role: LEADER
I20240502 08:04:00.957327 10585 consensus_queue.cc:237] T 385fa1c24869499ab21a9a6852f82494 P 9b42bd3f172344d59014088f13911462 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416a559822d4196a77a162a8bb09b99" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 34737 } } peers { permanent_uuid: "71013a2b500f4319bb04ffd933d71a8f" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 42107 } } peers { permanent_uuid: "9b42bd3f172344d59014088f13911462" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38051 } }
I20240502 08:04:00.963191 10283 catalog_manager.cc:5484] T 385fa1c24869499ab21a9a6852f82494 P 9b42bd3f172344d59014088f13911462 reported cstate change: term changed from 1 to 2, leader changed from 71013a2b500f4319bb04ffd933d71a8f (127.6.204.194) to 9b42bd3f172344d59014088f13911462 (127.6.204.195). New cstate: current_term: 2 leader_uuid: "9b42bd3f172344d59014088f13911462" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416a559822d4196a77a162a8bb09b99" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 34737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71013a2b500f4319bb04ffd933d71a8f" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 42107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b42bd3f172344d59014088f13911462" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38051 } health_report { overall_health: HEALTHY } } }
I20240502 08:04:01.253865 10461 raft_consensus.cc:1261] T 385fa1c24869499ab21a9a6852f82494 P 71013a2b500f4319bb04ffd933d71a8f [term 2 FOLLOWER]: Refusing update from remote peer 9b42bd3f172344d59014088f13911462: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:04:01.255192 10585 consensus_queue.cc:1035] T 385fa1c24869499ab21a9a6852f82494 P 9b42bd3f172344d59014088f13911462 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71013a2b500f4319bb04ffd933d71a8f" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 42107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:04:01.314340 10385 raft_consensus.cc:1261] T 385fa1c24869499ab21a9a6852f82494 P 9416a559822d4196a77a162a8bb09b99 [term 2 FOLLOWER]: Refusing update from remote peer 9b42bd3f172344d59014088f13911462: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:04:01.315600 10585 consensus_queue.cc:1035] T 385fa1c24869499ab21a9a6852f82494 P 9b42bd3f172344d59014088f13911462 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9416a559822d4196a77a162a8bb09b99" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 34737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:04:01.577462 10679 heartbeater.cc:503] Master 127.6.204.254:46313 was elected leader, sending a full tablet report...
I20240502 08:04:01.638329 10461 consensus_queue.cc:237] T b69b32e4d5b04f00956ef81511e7c9eb P 71013a2b500f4319bb04ffd933d71a8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9416a559822d4196a77a162a8bb09b99" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 34737 } } peers { permanent_uuid: "71013a2b500f4319bb04ffd933d71a8f" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 42107 } } peers { permanent_uuid: "9b42bd3f172344d59014088f13911462" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38051 } attrs { replace: true } } peers { permanent_uuid: "d202fa5d5d3a468483a3402915d7e8ac" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 41623 } attrs { promote: true } }
I20240502 08:04:01.645531 10535 raft_consensus.cc:1261] T b69b32e4d5b04f00956ef81511e7c9eb P 9b42bd3f172344d59014088f13911462 [term 1 FOLLOWER]: Refusing update from remote peer 71013a2b500f4319bb04ffd933d71a8f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:04:01.647739 10686 consensus_queue.cc:1035] T b69b32e4d5b04f00956ef81511e7c9eb P 71013a2b500f4319bb04ffd933d71a8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b42bd3f172344d59014088f13911462" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38051 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:04:01.649415 10385 raft_consensus.cc:1261] T b69b32e4d5b04f00956ef81511e7c9eb P 9416a559822d4196a77a162a8bb09b99 [term 1 FOLLOWER]: Refusing update from remote peer 71013a2b500f4319bb04ffd933d71a8f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:04:01.651088 10707 consensus_queue.cc:1035] T b69b32e4d5b04f00956ef81511e7c9eb P 71013a2b500f4319bb04ffd933d71a8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9416a559822d4196a77a162a8bb09b99" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 34737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 08:04:01.663587 10426 consensus_peers.cc:485] T b69b32e4d5b04f00956ef81511e7c9eb P 71013a2b500f4319bb04ffd933d71a8f -> Peer d202fa5d5d3a468483a3402915d7e8ac (127.6.204.196:41623): Couldn't send request to peer d202fa5d5d3a468483a3402915d7e8ac. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b69b32e4d5b04f00956ef81511e7c9eb. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:04:01.666710 10535 raft_consensus.cc:2932] T b69b32e4d5b04f00956ef81511e7c9eb P 9b42bd3f172344d59014088f13911462 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d202fa5d5d3a468483a3402915d7e8ac (127.6.204.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9416a559822d4196a77a162a8bb09b99" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 34737 } } peers { permanent_uuid: "71013a2b500f4319bb04ffd933d71a8f" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 42107 } } peers { permanent_uuid: "9b42bd3f172344d59014088f13911462" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38051 } attrs { replace: true } } peers { permanent_uuid: "d202fa5d5d3a468483a3402915d7e8ac" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 41623 } attrs { promote: true } } }
I20240502 08:04:01.667939 10385 raft_consensus.cc:2932] T b69b32e4d5b04f00956ef81511e7c9eb P 9416a559822d4196a77a162a8bb09b99 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d202fa5d5d3a468483a3402915d7e8ac (127.6.204.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9416a559822d4196a77a162a8bb09b99" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 34737 } } peers { permanent_uuid: "71013a2b500f4319bb04ffd933d71a8f" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 42107 } } peers { permanent_uuid: "9b42bd3f172344d59014088f13911462" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38051 } attrs { replace: true } } peers { permanent_uuid: "d202fa5d5d3a468483a3402915d7e8ac" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 41623 } attrs { promote: true } } }
I20240502 08:04:01.679971 10283 catalog_manager.cc:5484] T b69b32e4d5b04f00956ef81511e7c9eb P 9b42bd3f172344d59014088f13911462 reported cstate change: config changed from index -1 to 2, NON_VOTER d202fa5d5d3a468483a3402915d7e8ac (127.6.204.196) added. New cstate: current_term: 1 leader_uuid: "71013a2b500f4319bb04ffd933d71a8f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9416a559822d4196a77a162a8bb09b99" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 34737 } } peers { permanent_uuid: "71013a2b500f4319bb04ffd933d71a8f" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 42107 } } peers { permanent_uuid: "9b42bd3f172344d59014088f13911462" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38051 } attrs { replace: true } } peers { permanent_uuid: "d202fa5d5d3a468483a3402915d7e8ac" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 41623 } attrs { promote: true } } }
I20240502 08:04:01.804215 10707 raft_consensus.cc:2932] T b69b32e4d5b04f00956ef81511e7c9eb P 71013a2b500f4319bb04ffd933d71a8f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d202fa5d5d3a468483a3402915d7e8ac (127.6.204.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9416a559822d4196a77a162a8bb09b99" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 34737 } } peers { permanent_uuid: "71013a2b500f4319bb04ffd933d71a8f" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 42107 } } peers { permanent_uuid: "9b42bd3f172344d59014088f13911462" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38051 } attrs { replace: true } } peers { permanent_uuid: "d202fa5d5d3a468483a3402915d7e8ac" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 41623 } attrs { promote: true } } }
I20240502 08:04:01.825663 10534 consensus_queue.cc:237] T 385fa1c24869499ab21a9a6852f82494 P 9b42bd3f172344d59014088f13911462 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9416a559822d4196a77a162a8bb09b99" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 34737 } } peers { permanent_uuid: "71013a2b500f4319bb04ffd933d71a8f" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 42107 } attrs { replace: true } } peers { permanent_uuid: "9b42bd3f172344d59014088f13911462" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38051 } } peers { permanent_uuid: "d202fa5d5d3a468483a3402915d7e8ac" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 41623 } attrs { promote: true } }
I20240502 08:04:01.831599 10461 raft_consensus.cc:1261] T 385fa1c24869499ab21a9a6852f82494 P 71013a2b500f4319bb04ffd933d71a8f [term 2 FOLLOWER]: Refusing update from remote peer 9b42bd3f172344d59014088f13911462: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 08:04:01.832851 10385 raft_consensus.cc:1261] T 385fa1c24869499ab21a9a6852f82494 P 9416a559822d4196a77a162a8bb09b99 [term 2 FOLLOWER]: Refusing update from remote peer 9b42bd3f172344d59014088f13911462: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 08:04:01.832924 10585 consensus_queue.cc:1035] T 385fa1c24869499ab21a9a6852f82494 P 9b42bd3f172344d59014088f13911462 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71013a2b500f4319bb04ffd933d71a8f" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 42107 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 08:04:01.834497 10703 consensus_queue.cc:1035] T 385fa1c24869499ab21a9a6852f82494 P 9b42bd3f172344d59014088f13911462 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9416a559822d4196a77a162a8bb09b99" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 34737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240502 08:04:01.840837 10500 consensus_peers.cc:485] T 385fa1c24869499ab21a9a6852f82494 P 9b42bd3f172344d59014088f13911462 -> Peer d202fa5d5d3a468483a3402915d7e8ac (127.6.204.196:41623): Couldn't send request to peer d202fa5d5d3a468483a3402915d7e8ac. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 385fa1c24869499ab21a9a6852f82494. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:04:01.842371 10461 raft_consensus.cc:2932] T 385fa1c24869499ab21a9a6852f82494 P 71013a2b500f4319bb04ffd933d71a8f [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER d202fa5d5d3a468483a3402915d7e8ac (127.6.204.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9416a559822d4196a77a162a8bb09b99" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 34737 } } peers { permanent_uuid: "71013a2b500f4319bb04ffd933d71a8f" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 42107 } attrs { replace: true } } peers { permanent_uuid: "9b42bd3f172344d59014088f13911462" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38051 } } peers { permanent_uuid: "d202fa5d5d3a468483a3402915d7e8ac" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 41623 } attrs { promote: true } } }
I20240502 08:04:01.843868 10385 raft_consensus.cc:2932] T 385fa1c24869499ab21a9a6852f82494 P 9416a559822d4196a77a162a8bb09b99 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER d202fa5d5d3a468483a3402915d7e8ac (127.6.204.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9416a559822d4196a77a162a8bb09b99" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 34737 } } peers { permanent_uuid: "71013a2b500f4319bb04ffd933d71a8f" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 42107 } attrs { replace: true } } peers { permanent_uuid: "9b42bd3f172344d59014088f13911462" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38051 } } peers { permanent_uuid: "d202fa5d5d3a468483a3402915d7e8ac" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 41623 } attrs { promote: true } } }
I20240502 08:04:01.855368 10281 catalog_manager.cc:5484] T 385fa1c24869499ab21a9a6852f82494 P 71013a2b500f4319bb04ffd933d71a8f reported cstate change: config changed from index -1 to 3, NON_VOTER d202fa5d5d3a468483a3402915d7e8ac (127.6.204.196) added. New cstate: current_term: 2 leader_uuid: "9b42bd3f172344d59014088f13911462" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9416a559822d4196a77a162a8bb09b99" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 34737 } } peers { permanent_uuid: "71013a2b500f4319bb04ffd933d71a8f" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 42107 } attrs { replace: true } } peers { permanent_uuid: "9b42bd3f172344d59014088f13911462" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38051 } } peers { permanent_uuid: "d202fa5d5d3a468483a3402915d7e8ac" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 41623 } attrs { promote: true } } }
I20240502 08:04:01.910640 10703 raft_consensus.cc:2932] T 385fa1c24869499ab21a9a6852f82494 P 9b42bd3f172344d59014088f13911462 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER d202fa5d5d3a468483a3402915d7e8ac (127.6.204.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9416a559822d4196a77a162a8bb09b99" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 34737 } } peers { permanent_uuid: "71013a2b500f4319bb04ffd933d71a8f" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 42107 } attrs { replace: true } } peers { permanent_uuid: "9b42bd3f172344d59014088f13911462" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38051 } } peers { permanent_uuid: "d202fa5d5d3a468483a3402915d7e8ac" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 41623 } attrs { promote: true } } }
I20240502 08:04:01.993717 10719 ts_tablet_manager.cc:902] T b69b32e4d5b04f00956ef81511e7c9eb P d202fa5d5d3a468483a3402915d7e8ac: Initiating tablet copy from peer 71013a2b500f4319bb04ffd933d71a8f (127.6.204.194:42107)
I20240502 08:04:01.998219 10719 tablet_copy_client.cc:308] T b69b32e4d5b04f00956ef81511e7c9eb P d202fa5d5d3a468483a3402915d7e8ac: tablet copy: Beginning tablet copy session from remote peer at address 127.6.204.194:42107
I20240502 08:04:02.008666 10471 tablet_copy_service.cc:140] P 71013a2b500f4319bb04ffd933d71a8f: Received BeginTabletCopySession request for tablet b69b32e4d5b04f00956ef81511e7c9eb from peer d202fa5d5d3a468483a3402915d7e8ac ({username='slave'} at 127.0.0.1:46552)
I20240502 08:04:02.009295 10471 tablet_copy_service.cc:161] P 71013a2b500f4319bb04ffd933d71a8f: Beginning new tablet copy session on tablet b69b32e4d5b04f00956ef81511e7c9eb from peer d202fa5d5d3a468483a3402915d7e8ac at {username='slave'} at 127.0.0.1:46552: session id = d202fa5d5d3a468483a3402915d7e8ac-b69b32e4d5b04f00956ef81511e7c9eb
I20240502 08:04:02.016044 10471 tablet_copy_source_session.cc:205] T b69b32e4d5b04f00956ef81511e7c9eb P 71013a2b500f4319bb04ffd933d71a8f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:04:02.018867 10719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b69b32e4d5b04f00956ef81511e7c9eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:02.030705 10719 tablet_copy_client.cc:783] T b69b32e4d5b04f00956ef81511e7c9eb P d202fa5d5d3a468483a3402915d7e8ac: tablet copy: Starting download of 0 data blocks...
I20240502 08:04:02.031340 10719 tablet_copy_client.cc:647] T b69b32e4d5b04f00956ef81511e7c9eb P d202fa5d5d3a468483a3402915d7e8ac: tablet copy: Starting download of 1 WAL segments...
I20240502 08:04:02.035017 10719 tablet_copy_client.cc:523] T b69b32e4d5b04f00956ef81511e7c9eb P d202fa5d5d3a468483a3402915d7e8ac: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:04:02.043534 10719 tablet_bootstrap.cc:492] T b69b32e4d5b04f00956ef81511e7c9eb P d202fa5d5d3a468483a3402915d7e8ac: Bootstrap starting.
I20240502 08:04:02.061614 10719 tablet_bootstrap.cc:492] T b69b32e4d5b04f00956ef81511e7c9eb P d202fa5d5d3a468483a3402915d7e8ac: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:04:02.062361 10719 tablet_bootstrap.cc:492] T b69b32e4d5b04f00956ef81511e7c9eb P d202fa5d5d3a468483a3402915d7e8ac: Bootstrap complete.
I20240502 08:04:02.062912 10719 ts_tablet_manager.cc:1367] T b69b32e4d5b04f00956ef81511e7c9eb P d202fa5d5d3a468483a3402915d7e8ac: Time spent bootstrapping tablet: real 0.020s user 0.019s sys 0.000s
I20240502 08:04:02.065434 10719 raft_consensus.cc:348] T b69b32e4d5b04f00956ef81511e7c9eb P d202fa5d5d3a468483a3402915d7e8ac [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9416a559822d4196a77a162a8bb09b99" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 34737 } } peers { permanent_uuid: "71013a2b500f4319bb04ffd933d71a8f" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 42107 } } peers { permanent_uuid: "9b42bd3f172344d59014088f13911462" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38051 } attrs { replace: true } } peers { permanent_uuid: "d202fa5d5d3a468483a3402915d7e8ac" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 41623 } attrs { promote: true } }
I20240502 08:04:02.066169 10719 raft_consensus.cc:729] T b69b32e4d5b04f00956ef81511e7c9eb P d202fa5d5d3a468483a3402915d7e8ac [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d202fa5d5d3a468483a3402915d7e8ac, State: Initialized, Role: LEARNER
I20240502 08:04:02.066778 10719 consensus_queue.cc:260] T b69b32e4d5b04f00956ef81511e7c9eb P d202fa5d5d3a468483a3402915d7e8ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9416a559822d4196a77a162a8bb09b99" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 34737 } } peers { permanent_uuid: "71013a2b500f4319bb04ffd933d71a8f" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 42107 } } peers { permanent_uuid: "9b42bd3f172344d59014088f13911462" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38051 } attrs { replace: true } } peers { permanent_uuid: "d202fa5d5d3a468483a3402915d7e8ac" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 41623 } attrs { promote: true } }
I20240502 08:04:02.069778 10719 ts_tablet_manager.cc:1398] T b69b32e4d5b04f00956ef81511e7c9eb P d202fa5d5d3a468483a3402915d7e8ac: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240502 08:04:02.071537 10471 tablet_copy_service.cc:342] P 71013a2b500f4319bb04ffd933d71a8f: Request end of tablet copy session d202fa5d5d3a468483a3402915d7e8ac-b69b32e4d5b04f00956ef81511e7c9eb received from {username='slave'} at 127.0.0.1:46552
I20240502 08:04:02.071952 10471 tablet_copy_service.cc:433] P 71013a2b500f4319bb04ffd933d71a8f: ending tablet copy session d202fa5d5d3a468483a3402915d7e8ac-b69b32e4d5b04f00956ef81511e7c9eb on tablet b69b32e4d5b04f00956ef81511e7c9eb with peer d202fa5d5d3a468483a3402915d7e8ac
I20240502 08:04:02.190366 10719 ts_tablet_manager.cc:902] T 385fa1c24869499ab21a9a6852f82494 P d202fa5d5d3a468483a3402915d7e8ac: Initiating tablet copy from peer 9b42bd3f172344d59014088f13911462 (127.6.204.195:38051)
I20240502 08:04:02.194182 10719 tablet_copy_client.cc:308] T 385fa1c24869499ab21a9a6852f82494 P d202fa5d5d3a468483a3402915d7e8ac: tablet copy: Beginning tablet copy session from remote peer at address 127.6.204.195:38051
I20240502 08:04:02.203199 10545 tablet_copy_service.cc:140] P 9b42bd3f172344d59014088f13911462: Received BeginTabletCopySession request for tablet 385fa1c24869499ab21a9a6852f82494 from peer d202fa5d5d3a468483a3402915d7e8ac ({username='slave'} at 127.0.0.1:42840)
I20240502 08:04:02.203732 10545 tablet_copy_service.cc:161] P 9b42bd3f172344d59014088f13911462: Beginning new tablet copy session on tablet 385fa1c24869499ab21a9a6852f82494 from peer d202fa5d5d3a468483a3402915d7e8ac at {username='slave'} at 127.0.0.1:42840: session id = d202fa5d5d3a468483a3402915d7e8ac-385fa1c24869499ab21a9a6852f82494
I20240502 08:04:02.210026 10545 tablet_copy_source_session.cc:205] T 385fa1c24869499ab21a9a6852f82494 P 9b42bd3f172344d59014088f13911462: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:04:02.212697 10719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 385fa1c24869499ab21a9a6852f82494. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:02.219633 10653 raft_consensus.cc:1203] T b69b32e4d5b04f00956ef81511e7c9eb P d202fa5d5d3a468483a3402915d7e8ac [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:04:02.227365 10719 tablet_copy_client.cc:783] T 385fa1c24869499ab21a9a6852f82494 P d202fa5d5d3a468483a3402915d7e8ac: tablet copy: Starting download of 0 data blocks...
I20240502 08:04:02.228027 10719 tablet_copy_client.cc:647] T 385fa1c24869499ab21a9a6852f82494 P d202fa5d5d3a468483a3402915d7e8ac: tablet copy: Starting download of 1 WAL segments...
I20240502 08:04:02.231580 10719 tablet_copy_client.cc:523] T 385fa1c24869499ab21a9a6852f82494 P d202fa5d5d3a468483a3402915d7e8ac: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:04:02.239279 10719 tablet_bootstrap.cc:492] T 385fa1c24869499ab21a9a6852f82494 P d202fa5d5d3a468483a3402915d7e8ac: Bootstrap starting.
I20240502 08:04:02.259505 10719 tablet_bootstrap.cc:492] T 385fa1c24869499ab21a9a6852f82494 P d202fa5d5d3a468483a3402915d7e8ac: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:04:02.260416 10719 tablet_bootstrap.cc:492] T 385fa1c24869499ab21a9a6852f82494 P d202fa5d5d3a468483a3402915d7e8ac: Bootstrap complete.
I20240502 08:04:02.261101 10719 ts_tablet_manager.cc:1367] T 385fa1c24869499ab21a9a6852f82494 P d202fa5d5d3a468483a3402915d7e8ac: Time spent bootstrapping tablet: real 0.022s user 0.019s sys 0.000s
I20240502 08:04:02.263440 10719 raft_consensus.cc:348] T 385fa1c24869499ab21a9a6852f82494 P d202fa5d5d3a468483a3402915d7e8ac [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9416a559822d4196a77a162a8bb09b99" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 34737 } } peers { permanent_uuid: "71013a2b500f4319bb04ffd933d71a8f" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 42107 } attrs { replace: true } } peers { permanent_uuid: "9b42bd3f172344d59014088f13911462" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38051 } } peers { permanent_uuid: "d202fa5d5d3a468483a3402915d7e8ac" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 41623 } attrs { promote: true } }
I20240502 08:04:02.264204 10719 raft_consensus.cc:729] T 385fa1c24869499ab21a9a6852f82494 P d202fa5d5d3a468483a3402915d7e8ac [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: d202fa5d5d3a468483a3402915d7e8ac, State: Initialized, Role: LEARNER
I20240502 08:04:02.264801 10719 consensus_queue.cc:260] T 385fa1c24869499ab21a9a6852f82494 P d202fa5d5d3a468483a3402915d7e8ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9416a559822d4196a77a162a8bb09b99" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 34737 } } peers { permanent_uuid: "71013a2b500f4319bb04ffd933d71a8f" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 42107 } attrs { replace: true } } peers { permanent_uuid: "9b42bd3f172344d59014088f13911462" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38051 } } peers { permanent_uuid: "d202fa5d5d3a468483a3402915d7e8ac" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 41623 } attrs { promote: true } }
I20240502 08:04:02.266832 10719 ts_tablet_manager.cc:1398] T 385fa1c24869499ab21a9a6852f82494 P d202fa5d5d3a468483a3402915d7e8ac: Time spent starting tablet: real 0.005s user 0.004s sys 0.004s
I20240502 08:04:02.270298 10545 tablet_copy_service.cc:342] P 9b42bd3f172344d59014088f13911462: Request end of tablet copy session d202fa5d5d3a468483a3402915d7e8ac-385fa1c24869499ab21a9a6852f82494 received from {username='slave'} at 127.0.0.1:42840
I20240502 08:04:02.270738 10545 tablet_copy_service.cc:433] P 9b42bd3f172344d59014088f13911462: ending tablet copy session d202fa5d5d3a468483a3402915d7e8ac-385fa1c24869499ab21a9a6852f82494 on tablet 385fa1c24869499ab21a9a6852f82494 with peer d202fa5d5d3a468483a3402915d7e8ac
I20240502 08:04:02.475862 10653 raft_consensus.cc:1203] T 385fa1c24869499ab21a9a6852f82494 P d202fa5d5d3a468483a3402915d7e8ac [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20240502 08:04:02.607653 10337 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:04:02.611018 10337 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:04:02.611419 10337 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:04:02.736706 10707 raft_consensus.cc:1050] T b69b32e4d5b04f00956ef81511e7c9eb P 71013a2b500f4319bb04ffd933d71a8f: attempting to promote NON_VOTER d202fa5d5d3a468483a3402915d7e8ac to VOTER
I20240502 08:04:02.738785 10707 consensus_queue.cc:237] T b69b32e4d5b04f00956ef81511e7c9eb P 71013a2b500f4319bb04ffd933d71a8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9416a559822d4196a77a162a8bb09b99" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 34737 } } peers { permanent_uuid: "71013a2b500f4319bb04ffd933d71a8f" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 42107 } } peers { permanent_uuid: "9b42bd3f172344d59014088f13911462" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38051 } attrs { replace: true } } peers { permanent_uuid: "d202fa5d5d3a468483a3402915d7e8ac" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 41623 } attrs { promote: false } }
I20240502 08:04:02.744263 10653 raft_consensus.cc:1261] T b69b32e4d5b04f00956ef81511e7c9eb P d202fa5d5d3a468483a3402915d7e8ac [term 1 LEARNER]: Refusing update from remote peer 71013a2b500f4319bb04ffd933d71a8f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:04:02.744655 10534 raft_consensus.cc:1261] T b69b32e4d5b04f00956ef81511e7c9eb P 9b42bd3f172344d59014088f13911462 [term 1 FOLLOWER]: Refusing update from remote peer 71013a2b500f4319bb04ffd933d71a8f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:04:02.746057 10716 consensus_queue.cc:1035] T b69b32e4d5b04f00956ef81511e7c9eb P 71013a2b500f4319bb04ffd933d71a8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b42bd3f172344d59014088f13911462" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38051 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 08:04:02.747488 10732 consensus_queue.cc:1035] T b69b32e4d5b04f00956ef81511e7c9eb P 71013a2b500f4319bb04ffd933d71a8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d202fa5d5d3a468483a3402915d7e8ac" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 41623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 08:04:02.753448 10385 raft_consensus.cc:1261] T b69b32e4d5b04f00956ef81511e7c9eb P 9416a559822d4196a77a162a8bb09b99 [term 1 FOLLOWER]: Refusing update from remote peer 71013a2b500f4319bb04ffd933d71a8f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:04:02.756868 10732 consensus_queue.cc:1035] T b69b32e4d5b04f00956ef81511e7c9eb P 71013a2b500f4319bb04ffd933d71a8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9416a559822d4196a77a162a8bb09b99" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 34737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:04:02.762389 10653 raft_consensus.cc:2932] T b69b32e4d5b04f00956ef81511e7c9eb P d202fa5d5d3a468483a3402915d7e8ac [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d202fa5d5d3a468483a3402915d7e8ac (127.6.204.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9416a559822d4196a77a162a8bb09b99" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 34737 } } peers { permanent_uuid: "71013a2b500f4319bb04ffd933d71a8f" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 42107 } } peers { permanent_uuid: "9b42bd3f172344d59014088f13911462" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38051 } attrs { replace: true } } peers { permanent_uuid: "d202fa5d5d3a468483a3402915d7e8ac" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 41623 } attrs { promote: false } } }
I20240502 08:04:02.766968 10385 raft_consensus.cc:2932] T b69b32e4d5b04f00956ef81511e7c9eb P 9416a559822d4196a77a162a8bb09b99 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d202fa5d5d3a468483a3402915d7e8ac (127.6.204.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9416a559822d4196a77a162a8bb09b99" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 34737 } } peers { permanent_uuid: "71013a2b500f4319bb04ffd933d71a8f" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 42107 } } peers { permanent_uuid: "9b42bd3f172344d59014088f13911462" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38051 } attrs { replace: true } } peers { permanent_uuid: "d202fa5d5d3a468483a3402915d7e8ac" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 41623 } attrs { promote: false } } }
I20240502 08:04:02.777254 10284 catalog_manager.cc:5484] T b69b32e4d5b04f00956ef81511e7c9eb P d202fa5d5d3a468483a3402915d7e8ac reported cstate change: config changed from index 2 to 3, d202fa5d5d3a468483a3402915d7e8ac (127.6.204.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "71013a2b500f4319bb04ffd933d71a8f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9416a559822d4196a77a162a8bb09b99" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 34737 } } peers { permanent_uuid: "71013a2b500f4319bb04ffd933d71a8f" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 42107 } } peers { permanent_uuid: "9b42bd3f172344d59014088f13911462" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38051 } attrs { replace: true } } peers { permanent_uuid: "d202fa5d5d3a468483a3402915d7e8ac" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 41623 } attrs { promote: false } } }
I20240502 08:04:02.992501 10732 raft_consensus.cc:2932] T b69b32e4d5b04f00956ef81511e7c9eb P 71013a2b500f4319bb04ffd933d71a8f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d202fa5d5d3a468483a3402915d7e8ac (127.6.204.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9416a559822d4196a77a162a8bb09b99" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 34737 } } peers { permanent_uuid: "71013a2b500f4319bb04ffd933d71a8f" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 42107 } } peers { permanent_uuid: "9b42bd3f172344d59014088f13911462" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38051 } attrs { replace: true } } peers { permanent_uuid: "d202fa5d5d3a468483a3402915d7e8ac" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 41623 } attrs { promote: false } } }
I20240502 08:04:02.997112 10534 raft_consensus.cc:2932] T b69b32e4d5b04f00956ef81511e7c9eb P 9b42bd3f172344d59014088f13911462 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d202fa5d5d3a468483a3402915d7e8ac (127.6.204.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9416a559822d4196a77a162a8bb09b99" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 34737 } } peers { permanent_uuid: "71013a2b500f4319bb04ffd933d71a8f" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 42107 } } peers { permanent_uuid: "9b42bd3f172344d59014088f13911462" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38051 } attrs { replace: true } } peers { permanent_uuid: "d202fa5d5d3a468483a3402915d7e8ac" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 41623 } attrs { promote: false } } }
I20240502 08:04:03.011910 10461 consensus_queue.cc:237] T b69b32e4d5b04f00956ef81511e7c9eb P 71013a2b500f4319bb04ffd933d71a8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9416a559822d4196a77a162a8bb09b99" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 34737 } } peers { permanent_uuid: "71013a2b500f4319bb04ffd933d71a8f" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 42107 } } peers { permanent_uuid: "d202fa5d5d3a468483a3402915d7e8ac" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 41623 } attrs { promote: false } }
I20240502 08:04:03.015502 10585 raft_consensus.cc:1050] T 385fa1c24869499ab21a9a6852f82494 P 9b42bd3f172344d59014088f13911462: attempting to promote NON_VOTER d202fa5d5d3a468483a3402915d7e8ac to VOTER
I20240502 08:04:03.016889 10653 raft_consensus.cc:1261] T b69b32e4d5b04f00956ef81511e7c9eb P d202fa5d5d3a468483a3402915d7e8ac [term 1 FOLLOWER]: Refusing update from remote peer 71013a2b500f4319bb04ffd933d71a8f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:04:03.017025 10385 raft_consensus.cc:1261] T b69b32e4d5b04f00956ef81511e7c9eb P 9416a559822d4196a77a162a8bb09b99 [term 1 FOLLOWER]: Refusing update from remote peer 71013a2b500f4319bb04ffd933d71a8f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:04:03.018370 10707 consensus_queue.cc:1035] T b69b32e4d5b04f00956ef81511e7c9eb P 71013a2b500f4319bb04ffd933d71a8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d202fa5d5d3a468483a3402915d7e8ac" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 41623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:04:03.018085 10585 consensus_queue.cc:237] T 385fa1c24869499ab21a9a6852f82494 P 9b42bd3f172344d59014088f13911462 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9416a559822d4196a77a162a8bb09b99" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 34737 } } peers { permanent_uuid: "71013a2b500f4319bb04ffd933d71a8f" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 42107 } attrs { replace: true } } peers { permanent_uuid: "9b42bd3f172344d59014088f13911462" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38051 } } peers { permanent_uuid: "d202fa5d5d3a468483a3402915d7e8ac" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 41623 } attrs { promote: false } }
I20240502 08:04:03.019279 10716 consensus_queue.cc:1035] T b69b32e4d5b04f00956ef81511e7c9eb P 71013a2b500f4319bb04ffd933d71a8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9416a559822d4196a77a162a8bb09b99" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 34737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:04:03.028131 10653 raft_consensus.cc:2932] T b69b32e4d5b04f00956ef81511e7c9eb P d202fa5d5d3a468483a3402915d7e8ac [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9b42bd3f172344d59014088f13911462 (127.6.204.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9416a559822d4196a77a162a8bb09b99" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 34737 } } peers { permanent_uuid: "71013a2b500f4319bb04ffd933d71a8f" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 42107 } } peers { permanent_uuid: "d202fa5d5d3a468483a3402915d7e8ac" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 41623 } attrs { promote: false } } }
I20240502 08:04:03.028972 10652 raft_consensus.cc:1261] T 385fa1c24869499ab21a9a6852f82494 P d202fa5d5d3a468483a3402915d7e8ac [term 2 LEARNER]: Refusing update from remote peer 9b42bd3f172344d59014088f13911462: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 08:04:03.031064 10460 raft_consensus.cc:1261] T 385fa1c24869499ab21a9a6852f82494 P 71013a2b500f4319bb04ffd933d71a8f [term 2 FOLLOWER]: Refusing update from remote peer 9b42bd3f172344d59014088f13911462: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 08:04:03.031605 10385 raft_consensus.cc:1261] T 385fa1c24869499ab21a9a6852f82494 P 9416a559822d4196a77a162a8bb09b99 [term 2 FOLLOWER]: Refusing update from remote peer 9b42bd3f172344d59014088f13911462: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 08:04:03.032560 10703 consensus_queue.cc:1035] T 385fa1c24869499ab21a9a6852f82494 P 9b42bd3f172344d59014088f13911462 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d202fa5d5d3a468483a3402915d7e8ac" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 41623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:04:03.034435 10703 consensus_queue.cc:1035] T 385fa1c24869499ab21a9a6852f82494 P 9b42bd3f172344d59014088f13911462 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71013a2b500f4319bb04ffd933d71a8f" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 42107 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:04:03.035547 10585 consensus_queue.cc:1035] T 385fa1c24869499ab21a9a6852f82494 P 9b42bd3f172344d59014088f13911462 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9416a559822d4196a77a162a8bb09b99" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 34737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:04:03.039402 10385 raft_consensus.cc:2932] T b69b32e4d5b04f00956ef81511e7c9eb P 9416a559822d4196a77a162a8bb09b99 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9b42bd3f172344d59014088f13911462 (127.6.204.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9416a559822d4196a77a162a8bb09b99" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 34737 } } peers { permanent_uuid: "71013a2b500f4319bb04ffd933d71a8f" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 42107 } } peers { permanent_uuid: "d202fa5d5d3a468483a3402915d7e8ac" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 41623 } attrs { promote: false } } }
I20240502 08:04:03.051985 10283 catalog_manager.cc:5484] T b69b32e4d5b04f00956ef81511e7c9eb P 9416a559822d4196a77a162a8bb09b99 reported cstate change: config changed from index 3 to 4, VOTER 9b42bd3f172344d59014088f13911462 (127.6.204.195) evicted. New cstate: current_term: 1 leader_uuid: "71013a2b500f4319bb04ffd933d71a8f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9416a559822d4196a77a162a8bb09b99" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 34737 } } peers { permanent_uuid: "71013a2b500f4319bb04ffd933d71a8f" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 42107 } } peers { permanent_uuid: "d202fa5d5d3a468483a3402915d7e8ac" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 41623 } attrs { promote: false } } }
I20240502 08:04:03.071336 10386 raft_consensus.cc:2932] T 385fa1c24869499ab21a9a6852f82494 P 9416a559822d4196a77a162a8bb09b99 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, d202fa5d5d3a468483a3402915d7e8ac (127.6.204.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9416a559822d4196a77a162a8bb09b99" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 34737 } } peers { permanent_uuid: "71013a2b500f4319bb04ffd933d71a8f" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 42107 } attrs { replace: true } } peers { permanent_uuid: "9b42bd3f172344d59014088f13911462" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38051 } } peers { permanent_uuid: "d202fa5d5d3a468483a3402915d7e8ac" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 41623 } attrs { promote: false } } }
I20240502 08:04:03.072959 10460 raft_consensus.cc:2932] T 385fa1c24869499ab21a9a6852f82494 P 71013a2b500f4319bb04ffd933d71a8f [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, d202fa5d5d3a468483a3402915d7e8ac (127.6.204.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9416a559822d4196a77a162a8bb09b99" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 34737 } } peers { permanent_uuid: "71013a2b500f4319bb04ffd933d71a8f" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 42107 } attrs { replace: true } } peers { permanent_uuid: "9b42bd3f172344d59014088f13911462" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38051 } } peers { permanent_uuid: "d202fa5d5d3a468483a3402915d7e8ac" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 41623 } attrs { promote: false } } }
I20240502 08:04:03.086786 10281 catalog_manager.cc:5484] T 385fa1c24869499ab21a9a6852f82494 P 71013a2b500f4319bb04ffd933d71a8f reported cstate change: config changed from index 3 to 4, d202fa5d5d3a468483a3402915d7e8ac (127.6.204.196) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "9b42bd3f172344d59014088f13911462" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9416a559822d4196a77a162a8bb09b99" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 34737 } } peers { permanent_uuid: "71013a2b500f4319bb04ffd933d71a8f" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 42107 } attrs { replace: true } } peers { permanent_uuid: "9b42bd3f172344d59014088f13911462" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38051 } } peers { permanent_uuid: "d202fa5d5d3a468483a3402915d7e8ac" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 41623 } attrs { promote: false } } }
I20240502 08:04:03.213877 10585 raft_consensus.cc:2932] T 385fa1c24869499ab21a9a6852f82494 P 9b42bd3f172344d59014088f13911462 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, d202fa5d5d3a468483a3402915d7e8ac (127.6.204.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9416a559822d4196a77a162a8bb09b99" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 34737 } } peers { permanent_uuid: "71013a2b500f4319bb04ffd933d71a8f" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 42107 } attrs { replace: true } } peers { permanent_uuid: "9b42bd3f172344d59014088f13911462" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38051 } } peers { permanent_uuid: "d202fa5d5d3a468483a3402915d7e8ac" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 41623 } attrs { promote: false } } }
I20240502 08:04:03.223697 10653 raft_consensus.cc:2932] T 385fa1c24869499ab21a9a6852f82494 P d202fa5d5d3a468483a3402915d7e8ac [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, d202fa5d5d3a468483a3402915d7e8ac (127.6.204.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9416a559822d4196a77a162a8bb09b99" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 34737 } } peers { permanent_uuid: "71013a2b500f4319bb04ffd933d71a8f" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 42107 } attrs { replace: true } } peers { permanent_uuid: "9b42bd3f172344d59014088f13911462" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38051 } } peers { permanent_uuid: "d202fa5d5d3a468483a3402915d7e8ac" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 41623 } attrs { promote: false } } }
I20240502 08:04:03.280803 10524 tablet_service.cc:1514] Processing DeleteTablet for tablet b69b32e4d5b04f00956ef81511e7c9eb with delete_type TABLET_DATA_TOMBSTONED (TS 9b42bd3f172344d59014088f13911462 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:42788
I20240502 08:04:03.283120 10746 tablet_replica.cc:330] T b69b32e4d5b04f00956ef81511e7c9eb P 9b42bd3f172344d59014088f13911462: stopping tablet replica
I20240502 08:04:03.284070 10746 raft_consensus.cc:2229] T b69b32e4d5b04f00956ef81511e7c9eb P 9b42bd3f172344d59014088f13911462 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:04:03.284849 10746 raft_consensus.cc:2258] T b69b32e4d5b04f00956ef81511e7c9eb P 9b42bd3f172344d59014088f13911462 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:04:03.287802 10746 ts_tablet_manager.cc:1875] T b69b32e4d5b04f00956ef81511e7c9eb P 9b42bd3f172344d59014088f13911462: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:04:03.302487 10746 ts_tablet_manager.cc:1888] T b69b32e4d5b04f00956ef81511e7c9eb P 9b42bd3f172344d59014088f13911462: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 08:04:03.302853 10746 log.cc:1196] T b69b32e4d5b04f00956ef81511e7c9eb P 9b42bd3f172344d59014088f13911462: Deleting WAL directory at /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714636933625983-6963-0/minicluster-data/ts-2-root/wals/b69b32e4d5b04f00956ef81511e7c9eb
I20240502 08:04:03.304327 10271 catalog_manager.cc:4830] TS 9b42bd3f172344d59014088f13911462 (127.6.204.195:38051): tablet b69b32e4d5b04f00956ef81511e7c9eb (table test-workload [id=0640579302af4889b256057bbe62d105]) successfully deleted
I20240502 08:04:03.367777 10534 consensus_queue.cc:237] T 385fa1c24869499ab21a9a6852f82494 P 9b42bd3f172344d59014088f13911462 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9416a559822d4196a77a162a8bb09b99" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 34737 } } peers { permanent_uuid: "9b42bd3f172344d59014088f13911462" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38051 } } peers { permanent_uuid: "d202fa5d5d3a468483a3402915d7e8ac" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 41623 } attrs { promote: false } }
I20240502 08:04:03.373786 10653 raft_consensus.cc:1261] T 385fa1c24869499ab21a9a6852f82494 P d202fa5d5d3a468483a3402915d7e8ac [term 2 FOLLOWER]: Refusing update from remote peer 9b42bd3f172344d59014088f13911462: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 08:04:03.374815 10386 raft_consensus.cc:1261] T 385fa1c24869499ab21a9a6852f82494 P 9416a559822d4196a77a162a8bb09b99 [term 2 FOLLOWER]: Refusing update from remote peer 9b42bd3f172344d59014088f13911462: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 08:04:03.376801 10745 consensus_queue.cc:1035] T 385fa1c24869499ab21a9a6852f82494 P 9b42bd3f172344d59014088f13911462 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9416a559822d4196a77a162a8bb09b99" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 34737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240502 08:04:03.378005 10741 consensus_queue.cc:1035] T 385fa1c24869499ab21a9a6852f82494 P 9b42bd3f172344d59014088f13911462 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d202fa5d5d3a468483a3402915d7e8ac" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 41623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240502 08:04:03.387742 10653 raft_consensus.cc:2932] T 385fa1c24869499ab21a9a6852f82494 P d202fa5d5d3a468483a3402915d7e8ac [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 71013a2b500f4319bb04ffd933d71a8f (127.6.204.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9416a559822d4196a77a162a8bb09b99" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 34737 } } peers { permanent_uuid: "9b42bd3f172344d59014088f13911462" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38051 } } peers { permanent_uuid: "d202fa5d5d3a468483a3402915d7e8ac" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 41623 } attrs { promote: false } } }
I20240502 08:04:03.387502 10386 raft_consensus.cc:2932] T 385fa1c24869499ab21a9a6852f82494 P 9416a559822d4196a77a162a8bb09b99 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 71013a2b500f4319bb04ffd933d71a8f (127.6.204.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9416a559822d4196a77a162a8bb09b99" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 34737 } } peers { permanent_uuid: "9b42bd3f172344d59014088f13911462" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38051 } } peers { permanent_uuid: "d202fa5d5d3a468483a3402915d7e8ac" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 41623 } attrs { promote: false } } }
I20240502 08:04:03.400045 10283 catalog_manager.cc:5484] T 385fa1c24869499ab21a9a6852f82494 P 9416a559822d4196a77a162a8bb09b99 reported cstate change: config changed from index 4 to 5, VOTER 71013a2b500f4319bb04ffd933d71a8f (127.6.204.194) evicted. New cstate: current_term: 2 leader_uuid: "9b42bd3f172344d59014088f13911462" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9416a559822d4196a77a162a8bb09b99" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 34737 } } peers { permanent_uuid: "9b42bd3f172344d59014088f13911462" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38051 } } peers { permanent_uuid: "d202fa5d5d3a468483a3402915d7e8ac" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 41623 } attrs { promote: false } } }
I20240502 08:04:03.450418 10451 tablet_service.cc:1514] Processing DeleteTablet for tablet 385fa1c24869499ab21a9a6852f82494 with delete_type TABLET_DATA_TOMBSTONED (TS 71013a2b500f4319bb04ffd933d71a8f not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:46516
I20240502 08:04:03.452155 10747 tablet_replica.cc:330] T 385fa1c24869499ab21a9a6852f82494 P 71013a2b500f4319bb04ffd933d71a8f: stopping tablet replica
I20240502 08:04:03.452926 10747 raft_consensus.cc:2229] T 385fa1c24869499ab21a9a6852f82494 P 71013a2b500f4319bb04ffd933d71a8f [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:04:03.453667 10747 raft_consensus.cc:2258] T 385fa1c24869499ab21a9a6852f82494 P 71013a2b500f4319bb04ffd933d71a8f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:04:03.456753 10747 ts_tablet_manager.cc:1875] T 385fa1c24869499ab21a9a6852f82494 P 71013a2b500f4319bb04ffd933d71a8f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:04:03.473138 10747 ts_tablet_manager.cc:1888] T 385fa1c24869499ab21a9a6852f82494 P 71013a2b500f4319bb04ffd933d71a8f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240502 08:04:03.473562 10747 log.cc:1196] T 385fa1c24869499ab21a9a6852f82494 P 71013a2b500f4319bb04ffd933d71a8f: Deleting WAL directory at /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714636933625983-6963-0/minicluster-data/ts-1-root/wals/385fa1c24869499ab21a9a6852f82494
I20240502 08:04:03.475705 10269 catalog_manager.cc:4830] TS 71013a2b500f4319bb04ffd933d71a8f (127.6.204.194:42107): tablet 385fa1c24869499ab21a9a6852f82494 (table test-workload [id=0640579302af4889b256057bbe62d105]) successfully deleted
I20240502 08:04:03.726317 10732 raft_consensus.cc:2932] T b69b32e4d5b04f00956ef81511e7c9eb P 71013a2b500f4319bb04ffd933d71a8f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9b42bd3f172344d59014088f13911462 (127.6.204.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9416a559822d4196a77a162a8bb09b99" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 34737 } } peers { permanent_uuid: "71013a2b500f4319bb04ffd933d71a8f" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 42107 } } peers { permanent_uuid: "d202fa5d5d3a468483a3402915d7e8ac" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 41623 } attrs { promote: false } } }
I20240502 08:04:03.733870 10269 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b69b32e4d5b04f00956ef81511e7c9eb with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:04:04.049822 10585 raft_consensus.cc:2932] T 385fa1c24869499ab21a9a6852f82494 P 9b42bd3f172344d59014088f13911462 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 71013a2b500f4319bb04ffd933d71a8f (127.6.204.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9416a559822d4196a77a162a8bb09b99" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 34737 } } peers { permanent_uuid: "9b42bd3f172344d59014088f13911462" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 38051 } } peers { permanent_uuid: "d202fa5d5d3a468483a3402915d7e8ac" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 41623 } attrs { promote: false } } }
I20240502 08:04:04.056914 10271 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 385fa1c24869499ab21a9a6852f82494 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:04:04.612695 10337 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:04:04.615901 10337 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:04:04.616246 10337 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:04:06.617234 10337 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:04:06.620515 10337 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:04:06.620975 10337 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:04:08.622023 10337 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:04:08.624640 10337 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:04:08.624986 10337 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:04:10.625893 10337 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:04:10.628527 10337 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:04:10.628885 10337 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:04:10.701961 6963 tablet_server.cc:178] TabletServer@127.6.204.193:0 shutting down...
I20240502 08:04:10.732875 6963 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 08:04:10.733655 6963 tablet_replica.cc:330] T b69b32e4d5b04f00956ef81511e7c9eb P 9416a559822d4196a77a162a8bb09b99: stopping tablet replica
I20240502 08:04:10.734341 6963 raft_consensus.cc:2229] T b69b32e4d5b04f00956ef81511e7c9eb P 9416a559822d4196a77a162a8bb09b99 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:04:10.735390 6963 raft_consensus.cc:2258] T b69b32e4d5b04f00956ef81511e7c9eb P 9416a559822d4196a77a162a8bb09b99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:04:10.737929 6963 tablet_replica.cc:330] T 434c464e11b64a00ae9d3d3097ca345d P 9416a559822d4196a77a162a8bb09b99: stopping tablet replica
I20240502 08:04:10.738467 6963 raft_consensus.cc:2229] T 434c464e11b64a00ae9d3d3097ca345d P 9416a559822d4196a77a162a8bb09b99 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:04:10.739332 6963 raft_consensus.cc:2258] T 434c464e11b64a00ae9d3d3097ca345d P 9416a559822d4196a77a162a8bb09b99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:04:10.741528 6963 tablet_replica.cc:330] T 385fa1c24869499ab21a9a6852f82494 P 9416a559822d4196a77a162a8bb09b99: stopping tablet replica
I20240502 08:04:10.742012 6963 raft_consensus.cc:2229] T 385fa1c24869499ab21a9a6852f82494 P 9416a559822d4196a77a162a8bb09b99 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:04:10.742985 6963 raft_consensus.cc:2258] T 385fa1c24869499ab21a9a6852f82494 P 9416a559822d4196a77a162a8bb09b99 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:04:10.767233 6963 tablet_server.cc:194] TabletServer@127.6.204.193:0 shutdown complete.
I20240502 08:04:10.782327 6963 tablet_server.cc:178] TabletServer@127.6.204.194:0 shutting down...
I20240502 08:04:10.809031 6963 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 08:04:10.809752 6963 tablet_replica.cc:330] T 434c464e11b64a00ae9d3d3097ca345d P 71013a2b500f4319bb04ffd933d71a8f: stopping tablet replica
I20240502 08:04:10.810405 6963 raft_consensus.cc:2229] T 434c464e11b64a00ae9d3d3097ca345d P 71013a2b500f4319bb04ffd933d71a8f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:04:10.810998 6963 raft_consensus.cc:2258] T 434c464e11b64a00ae9d3d3097ca345d P 71013a2b500f4319bb04ffd933d71a8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:04:10.813416 6963 tablet_replica.cc:330] T b69b32e4d5b04f00956ef81511e7c9eb P 71013a2b500f4319bb04ffd933d71a8f: stopping tablet replica
I20240502 08:04:10.813961 6963 raft_consensus.cc:2229] T b69b32e4d5b04f00956ef81511e7c9eb P 71013a2b500f4319bb04ffd933d71a8f [term 1 LEADER]: Raft consensus shutting down.
W20240502 08:04:10.814226 10500 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.204.193:34737: connect: Connection refused (error 111)
I20240502 08:04:10.815012 6963 raft_consensus.cc:2258] T b69b32e4d5b04f00956ef81511e7c9eb P 71013a2b500f4319bb04ffd933d71a8f [term 1 FOLLOWER]: Raft consensus is shut down!
W20240502 08:04:10.817941 10500 consensus_peers.cc:485] T 385fa1c24869499ab21a9a6852f82494 P 9b42bd3f172344d59014088f13911462 -> Peer 9416a559822d4196a77a162a8bb09b99 (127.6.204.193:34737): Couldn't send request to peer 9416a559822d4196a77a162a8bb09b99. Status: Network error: Client connection negotiation failed: client connection to 127.6.204.193:34737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:04:10.840101 6963 tablet_server.cc:194] TabletServer@127.6.204.194:0 shutdown complete.
I20240502 08:04:10.854512 6963 tablet_server.cc:178] TabletServer@127.6.204.195:0 shutting down...
I20240502 08:04:10.879431 6963 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 08:04:10.880275 6963 tablet_replica.cc:330] T 385fa1c24869499ab21a9a6852f82494 P 9b42bd3f172344d59014088f13911462: stopping tablet replica
I20240502 08:04:10.880870 6963 raft_consensus.cc:2229] T 385fa1c24869499ab21a9a6852f82494 P 9b42bd3f172344d59014088f13911462 [term 2 LEADER]: Raft consensus shutting down.
I20240502 08:04:10.881713 6963 raft_consensus.cc:2258] T 385fa1c24869499ab21a9a6852f82494 P 9b42bd3f172344d59014088f13911462 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:04:10.883714 6963 tablet_replica.cc:330] T 434c464e11b64a00ae9d3d3097ca345d P 9b42bd3f172344d59014088f13911462: stopping tablet replica
I20240502 08:04:10.884153 6963 raft_consensus.cc:2229] T 434c464e11b64a00ae9d3d3097ca345d P 9b42bd3f172344d59014088f13911462 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:04:10.884531 6963 raft_consensus.cc:2258] T 434c464e11b64a00ae9d3d3097ca345d P 9b42bd3f172344d59014088f13911462 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:04:10.908912 6963 tablet_server.cc:194] TabletServer@127.6.204.195:0 shutdown complete.
I20240502 08:04:10.923189 6963 tablet_server.cc:178] TabletServer@127.6.204.196:0 shutting down...
I20240502 08:04:10.944085 6963 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 08:04:10.944864 6963 tablet_replica.cc:330] T 385fa1c24869499ab21a9a6852f82494 P d202fa5d5d3a468483a3402915d7e8ac: stopping tablet replica
I20240502 08:04:10.945510 6963 raft_consensus.cc:2229] T 385fa1c24869499ab21a9a6852f82494 P d202fa5d5d3a468483a3402915d7e8ac [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:04:10.946202 6963 raft_consensus.cc:2258] T 385fa1c24869499ab21a9a6852f82494 P d202fa5d5d3a468483a3402915d7e8ac [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:04:10.948153 6963 tablet_replica.cc:330] T b69b32e4d5b04f00956ef81511e7c9eb P d202fa5d5d3a468483a3402915d7e8ac: stopping tablet replica
I20240502 08:04:10.948596 6963 raft_consensus.cc:2229] T b69b32e4d5b04f00956ef81511e7c9eb P d202fa5d5d3a468483a3402915d7e8ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:04:10.949235 6963 raft_consensus.cc:2258] T b69b32e4d5b04f00956ef81511e7c9eb P d202fa5d5d3a468483a3402915d7e8ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:04:10.969558 6963 tablet_server.cc:194] TabletServer@127.6.204.196:0 shutdown complete.
I20240502 08:04:10.981806 6963 master.cc:537] Master@127.6.204.254:46313 shutting down...
I20240502 08:04:11.002007 6963 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 93264b44a7ca4f0f82ac003ffae0d042 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:04:11.002627 6963 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 93264b44a7ca4f0f82ac003ffae0d042 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:04:11.002938 6963 tablet_replica.cc:330] T 00000000000000000000000000000000 P 93264b44a7ca4f0f82ac003ffae0d042: stopping tablet replica
I20240502 08:04:11.020864 6963 master.cc:552] Master@127.6.204.254:46313 shutdown complete.
[ OK ] AutoRebalancerTest.AutoRebalancingUnstableCluster (12620 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor
I20240502 08:04:11.061508 6963 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.6.204.254:43795
I20240502 08:04:11.062673 6963 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:04:11.068920 10753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:04:11.069038 10754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:04:11.069375 10756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:04:11.070672 6963 server_base.cc:1034] running on GCE node
I20240502 08:04:11.071465 6963 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:04:11.071660 6963 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:04:11.071797 6963 hybrid_clock.cc:648] HybridClock initialized: now 1714637051071780 us; error 0 us; skew 500 ppm
I20240502 08:04:11.072317 6963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:04:11.080699 6963 webserver.cc:460] Webserver started at http://127.6.204.254:43479/ using document root <none> and password file <none>
I20240502 08:04:11.081187 6963 fs_manager.cc:359] Metadata directory not provided
I20240502 08:04:11.081355 6963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:04:11.081635 6963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:04:11.082911 6963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714636933625983-6963-0/minicluster-data/master-0-root/instance:
uuid: "68987d3adb6f4490ae2f655c7a141fcd"
format_stamp: "Formatted at 2024-05-02 08:04:11 on dist-test-slave-32xs"
I20240502 08:04:11.087374 6963 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 08:04:11.090703 10761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:04:11.091434 6963 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 08:04:11.091714 6963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714636933625983-6963-0/minicluster-data/master-0-root
uuid: "68987d3adb6f4490ae2f655c7a141fcd"
format_stamp: "Formatted at 2024-05-02 08:04:11 on dist-test-slave-32xs"
I20240502 08:04:11.091986 6963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714636933625983-6963-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714636933625983-6963-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714636933625983-6963-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:04:11.113101 6963 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:04:11.114425 6963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:04:11.151392 6963 rpc_server.cc:307] RPC server started. Bound to: 127.6.204.254:43795
I20240502 08:04:11.151494 10812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.204.254:43795 every 8 connection(s)
I20240502 08:04:11.155265 10813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:11.165756 10813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68987d3adb6f4490ae2f655c7a141fcd: Bootstrap starting.
I20240502 08:04:11.169988 10813 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68987d3adb6f4490ae2f655c7a141fcd: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:11.174080 10813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68987d3adb6f4490ae2f655c7a141fcd: No bootstrap required, opened a new log
I20240502 08:04:11.176045 10813 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68987d3adb6f4490ae2f655c7a141fcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68987d3adb6f4490ae2f655c7a141fcd" member_type: VOTER }
I20240502 08:04:11.176476 10813 raft_consensus.cc:374] T 00000000000000000000000000000000 P 68987d3adb6f4490ae2f655c7a141fcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:11.176653 10813 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68987d3adb6f4490ae2f655c7a141fcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68987d3adb6f4490ae2f655c7a141fcd, State: Initialized, Role: FOLLOWER
I20240502 08:04:11.177130 10813 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68987d3adb6f4490ae2f655c7a141fcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68987d3adb6f4490ae2f655c7a141fcd" member_type: VOTER }
I20240502 08:04:11.177553 10813 raft_consensus.cc:388] T 00000000000000000000000000000000 P 68987d3adb6f4490ae2f655c7a141fcd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:04:11.177744 10813 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68987d3adb6f4490ae2f655c7a141fcd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:04:11.177954 10813 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68987d3adb6f4490ae2f655c7a141fcd [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:11.182286 10813 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68987d3adb6f4490ae2f655c7a141fcd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68987d3adb6f4490ae2f655c7a141fcd" member_type: VOTER }
I20240502 08:04:11.182750 10813 leader_election.cc:302] T 00000000000000000000000000000000 P 68987d3adb6f4490ae2f655c7a141fcd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 68987d3adb6f4490ae2f655c7a141fcd; no voters:
I20240502 08:04:11.183837 10813 leader_election.cc:288] T 00000000000000000000000000000000 P 68987d3adb6f4490ae2f655c7a141fcd [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:04:11.184154 10816 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 68987d3adb6f4490ae2f655c7a141fcd [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:04:11.185439 10816 raft_consensus.cc:686] T 00000000000000000000000000000000 P 68987d3adb6f4490ae2f655c7a141fcd [term 1 LEADER]: Becoming Leader. State: Replica: 68987d3adb6f4490ae2f655c7a141fcd, State: Running, Role: LEADER
I20240502 08:04:11.186065 10816 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68987d3adb6f4490ae2f655c7a141fcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68987d3adb6f4490ae2f655c7a141fcd" member_type: VOTER }
I20240502 08:04:11.186808 10813 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68987d3adb6f4490ae2f655c7a141fcd [sys.catalog]: configured and running, proceeding with master startup.
I20240502 08:04:11.191190 10818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68987d3adb6f4490ae2f655c7a141fcd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68987d3adb6f4490ae2f655c7a141fcd. Latest consensus state: current_term: 1 leader_uuid: "68987d3adb6f4490ae2f655c7a141fcd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68987d3adb6f4490ae2f655c7a141fcd" member_type: VOTER } }
I20240502 08:04:11.192020 10817 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68987d3adb6f4490ae2f655c7a141fcd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "68987d3adb6f4490ae2f655c7a141fcd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68987d3adb6f4490ae2f655c7a141fcd" member_type: VOTER } }
I20240502 08:04:11.192580 10818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68987d3adb6f4490ae2f655c7a141fcd [sys.catalog]: This master's current role is: LEADER
I20240502 08:04:11.192813 10817 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68987d3adb6f4490ae2f655c7a141fcd [sys.catalog]: This master's current role is: LEADER
I20240502 08:04:11.194834 10823 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 08:04:11.199985 10823 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 08:04:11.202801 6963 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 08:04:11.208825 10823 catalog_manager.cc:1347] Generated new cluster ID: c2df921aa19240288bb847997c39e836
I20240502 08:04:11.209085 10823 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 08:04:11.227273 10823 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 08:04:11.228538 10823 catalog_manager.cc:1504] Loading token signing keys...
I20240502 08:04:11.242628 10823 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 68987d3adb6f4490ae2f655c7a141fcd: Generated new TSK 0
I20240502 08:04:11.243257 10823 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 08:04:11.269811 6963 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:04:11.275561 10834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:04:11.276882 10835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:04:11.278898 10837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:04:11.279001 6963 server_base.cc:1034] running on GCE node
I20240502 08:04:11.279986 6963 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:04:11.280179 6963 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:04:11.280340 6963 hybrid_clock.cc:648] HybridClock initialized: now 1714637051280321 us; error 0 us; skew 500 ppm
I20240502 08:04:11.280887 6963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:04:11.283318 6963 webserver.cc:460] Webserver started at http://127.6.204.193:37609/ using document root <none> and password file <none>
I20240502 08:04:11.283789 6963 fs_manager.cc:359] Metadata directory not provided
I20240502 08:04:11.283974 6963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:04:11.284227 6963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:04:11.285374 6963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714636933625983-6963-0/minicluster-data/ts-0-root/instance:
uuid: "9e8f879cc8c44deba93997cc70ce375b"
format_stamp: "Formatted at 2024-05-02 08:04:11 on dist-test-slave-32xs"
I20240502 08:04:11.290108 6963 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20240502 08:04:11.293632 10842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:04:11.294514 6963 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20240502 08:04:11.294783 6963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714636933625983-6963-0/minicluster-data/ts-0-root
uuid: "9e8f879cc8c44deba93997cc70ce375b"
format_stamp: "Formatted at 2024-05-02 08:04:11 on dist-test-slave-32xs"
I20240502 08:04:11.295060 6963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714636933625983-6963-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714636933625983-6963-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714636933625983-6963-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:04:11.345778 6963 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:04:11.347002 6963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:04:11.348443 6963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:04:11.351141 6963 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:04:11.351333 6963 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:04:11.351560 6963 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:04:11.351727 6963 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:04:11.391212 6963 rpc_server.cc:307] RPC server started. Bound to: 127.6.204.193:46505
I20240502 08:04:11.391311 10904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.204.193:46505 every 8 connection(s)
I20240502 08:04:11.396047 6963 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:04:11.404179 10909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:04:11.404717 10910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:04:11.409050 6963 server_base.cc:1034] running on GCE node
W20240502 08:04:11.409235 10912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:04:11.410060 10905 heartbeater.cc:347] Connected to a master server at 127.6.204.254:43795
I20240502 08:04:11.410246 6963 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:04:11.410534 6963 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:04:11.410647 10905 heartbeater.cc:464] Registering TS with master...
I20240502 08:04:11.410691 6963 hybrid_clock.cc:648] HybridClock initialized: now 1714637051410674 us; error 0 us; skew 500 ppm
I20240502 08:04:11.411466 6963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:04:11.411590 10905 heartbeater.cc:511] Master 127.6.204.254:43795 requested a full tablet report, sending...
I20240502 08:04:11.413748 10778 ts_manager.cc:194] Registered new tserver with Master: 9e8f879cc8c44deba93997cc70ce375b (127.6.204.193:46505)
I20240502 08:04:11.414314 6963 webserver.cc:460] Webserver started at http://127.6.204.194:43647/ using document root <none> and password file <none>
I20240502 08:04:11.414808 6963 fs_manager.cc:359] Metadata directory not provided
I20240502 08:04:11.414997 6963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:04:11.415262 6963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:04:11.415771 10778 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51786
I20240502 08:04:11.416489 6963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714636933625983-6963-0/minicluster-data/ts-1-root/instance:
uuid: "3378106d9722470cb179d01d8e5c9160"
format_stamp: "Formatted at 2024-05-02 08:04:11 on dist-test-slave-32xs"
I20240502 08:04:11.421043 6963 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240502 08:04:11.424270 10917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:04:11.425040 6963 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20240502 08:04:11.425305 6963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714636933625983-6963-0/minicluster-data/ts-1-root
uuid: "3378106d9722470cb179d01d8e5c9160"
format_stamp: "Formatted at 2024-05-02 08:04:11 on dist-test-slave-32xs"
I20240502 08:04:11.425577 6963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714636933625983-6963-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714636933625983-6963-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714636933625983-6963-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:04:11.442749 6963 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:04:11.443926 6963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:04:11.445339 6963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:04:11.447713 6963 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:04:11.447911 6963 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:04:11.448134 6963 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:04:11.448285 6963 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:04:11.488174 6963 rpc_server.cc:307] RPC server started. Bound to: 127.6.204.194:44223
I20240502 08:04:11.488302 10979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.204.194:44223 every 8 connection(s)
I20240502 08:04:11.492846 6963 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:04:11.499608 10983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:04:11.500238 10984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:04:11.503388 6963 server_base.cc:1034] running on GCE node
W20240502 08:04:11.504588 10987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:04:11.504917 10980 heartbeater.cc:347] Connected to a master server at 127.6.204.254:43795
I20240502 08:04:11.505374 10980 heartbeater.cc:464] Registering TS with master...
I20240502 08:04:11.505636 6963 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:04:11.505905 6963 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:04:11.506110 6963 hybrid_clock.cc:648] HybridClock initialized: now 1714637051506092 us; error 0 us; skew 500 ppm
I20240502 08:04:11.506327 10980 heartbeater.cc:511] Master 127.6.204.254:43795 requested a full tablet report, sending...
I20240502 08:04:11.506897 6963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:04:11.508390 10778 ts_manager.cc:194] Registered new tserver with Master: 3378106d9722470cb179d01d8e5c9160 (127.6.204.194:44223)
I20240502 08:04:11.509775 10778 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51792
I20240502 08:04:11.510090 6963 webserver.cc:460] Webserver started at http://127.6.204.195:35551/ using document root <none> and password file <none>
I20240502 08:04:11.510651 6963 fs_manager.cc:359] Metadata directory not provided
I20240502 08:04:11.510848 6963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:04:11.511178 6963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:04:11.512718 6963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714636933625983-6963-0/minicluster-data/ts-2-root/instance:
uuid: "0b9a43cf64e54f909cfe47c7f4834ee8"
format_stamp: "Formatted at 2024-05-02 08:04:11 on dist-test-slave-32xs"
I20240502 08:04:11.517114 6963 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240502 08:04:11.520426 10991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:04:11.521199 6963 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 08:04:11.521485 6963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714636933625983-6963-0/minicluster-data/ts-2-root
uuid: "0b9a43cf64e54f909cfe47c7f4834ee8"
format_stamp: "Formatted at 2024-05-02 08:04:11 on dist-test-slave-32xs"
I20240502 08:04:11.521764 6963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714636933625983-6963-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714636933625983-6963-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714636933625983-6963-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:04:11.532763 6963 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:04:11.533911 6963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:04:11.535387 6963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:04:11.537667 6963 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:04:11.537876 6963 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:04:11.538105 6963 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:04:11.538317 6963 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20240502 08:04:11.577843 6963 rpc_server.cc:307] RPC server started. Bound to: 127.6.204.195:36523
I20240502 08:04:11.577942 11053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.204.195:36523 every 8 connection(s)
I20240502 08:04:11.591394 11054 heartbeater.cc:347] Connected to a master server at 127.6.204.254:43795
I20240502 08:04:11.591743 11054 heartbeater.cc:464] Registering TS with master...
I20240502 08:04:11.592414 11054 heartbeater.cc:511] Master 127.6.204.254:43795 requested a full tablet report, sending...
I20240502 08:04:11.594220 10778 ts_manager.cc:194] Registered new tserver with Master: 0b9a43cf64e54f909cfe47c7f4834ee8 (127.6.204.195:36523)
I20240502 08:04:11.594616 6963 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013399135s
I20240502 08:04:11.595619 10778 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51798
I20240502 08:04:12.418440 10905 heartbeater.cc:503] Master 127.6.204.254:43795 was elected leader, sending a full tablet report...
I20240502 08:04:12.512581 10980 heartbeater.cc:503] Master 127.6.204.254:43795 was elected leader, sending a full tablet report...
I20240502 08:04:12.598181 11054 heartbeater.cc:503] Master 127.6.204.254:43795 was elected leader, sending a full tablet report...
I20240502 08:04:12.627648 6963 test_util.cc:268] Using random seed: 208718505
I20240502 08:04:12.651751 10778 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51812:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:04:12.653699 10778 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:04:12.703178 10945 tablet_service.cc:1467] Processing CreateTablet for tablet 3c700781a5304f31a7ede719eea1ae9a (DEFAULT_TABLE table=test-workload [id=3bcf08a8ea5846f29ada278e310060cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:04:12.703434 10870 tablet_service.cc:1467] Processing CreateTablet for tablet 3c700781a5304f31a7ede719eea1ae9a (DEFAULT_TABLE table=test-workload [id=3bcf08a8ea5846f29ada278e310060cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:04:12.704587 10945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c700781a5304f31a7ede719eea1ae9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:12.704861 10870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c700781a5304f31a7ede719eea1ae9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:12.713867 11019 tablet_service.cc:1467] Processing CreateTablet for tablet 3c700781a5304f31a7ede719eea1ae9a (DEFAULT_TABLE table=test-workload [id=3bcf08a8ea5846f29ada278e310060cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:04:12.715184 11019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c700781a5304f31a7ede719eea1ae9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:12.719631 11074 tablet_bootstrap.cc:492] T 3c700781a5304f31a7ede719eea1ae9a P 3378106d9722470cb179d01d8e5c9160: Bootstrap starting.
I20240502 08:04:12.721980 11075 tablet_bootstrap.cc:492] T 3c700781a5304f31a7ede719eea1ae9a P 9e8f879cc8c44deba93997cc70ce375b: Bootstrap starting.
I20240502 08:04:12.728299 11075 tablet_bootstrap.cc:654] T 3c700781a5304f31a7ede719eea1ae9a P 9e8f879cc8c44deba93997cc70ce375b: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:12.728308 11074 tablet_bootstrap.cc:654] T 3c700781a5304f31a7ede719eea1ae9a P 3378106d9722470cb179d01d8e5c9160: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:12.730386 11076 tablet_bootstrap.cc:492] T 3c700781a5304f31a7ede719eea1ae9a P 0b9a43cf64e54f909cfe47c7f4834ee8: Bootstrap starting.
I20240502 08:04:12.736317 11076 tablet_bootstrap.cc:654] T 3c700781a5304f31a7ede719eea1ae9a P 0b9a43cf64e54f909cfe47c7f4834ee8: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:12.738610 11074 tablet_bootstrap.cc:492] T 3c700781a5304f31a7ede719eea1ae9a P 3378106d9722470cb179d01d8e5c9160: No bootstrap required, opened a new log
I20240502 08:04:12.739048 11075 tablet_bootstrap.cc:492] T 3c700781a5304f31a7ede719eea1ae9a P 9e8f879cc8c44deba93997cc70ce375b: No bootstrap required, opened a new log
I20240502 08:04:12.739181 11074 ts_tablet_manager.cc:1367] T 3c700781a5304f31a7ede719eea1ae9a P 3378106d9722470cb179d01d8e5c9160: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.013s
I20240502 08:04:12.739511 11075 ts_tablet_manager.cc:1367] T 3c700781a5304f31a7ede719eea1ae9a P 9e8f879cc8c44deba93997cc70ce375b: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.012s
I20240502 08:04:12.741791 11076 tablet_bootstrap.cc:492] T 3c700781a5304f31a7ede719eea1ae9a P 0b9a43cf64e54f909cfe47c7f4834ee8: No bootstrap required, opened a new log
I20240502 08:04:12.741839 11075 raft_consensus.cc:348] T 3c700781a5304f31a7ede719eea1ae9a P 9e8f879cc8c44deba93997cc70ce375b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3378106d9722470cb179d01d8e5c9160" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44223 } } peers { permanent_uuid: "9e8f879cc8c44deba93997cc70ce375b" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 46505 } } peers { permanent_uuid: "0b9a43cf64e54f909cfe47c7f4834ee8" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 36523 } }
I20240502 08:04:12.742559 11076 ts_tablet_manager.cc:1367] T 3c700781a5304f31a7ede719eea1ae9a P 0b9a43cf64e54f909cfe47c7f4834ee8: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.003s
I20240502 08:04:12.742672 11075 raft_consensus.cc:374] T 3c700781a5304f31a7ede719eea1ae9a P 9e8f879cc8c44deba93997cc70ce375b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:12.742297 11074 raft_consensus.cc:348] T 3c700781a5304f31a7ede719eea1ae9a P 3378106d9722470cb179d01d8e5c9160 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3378106d9722470cb179d01d8e5c9160" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44223 } } peers { permanent_uuid: "9e8f879cc8c44deba93997cc70ce375b" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 46505 } } peers { permanent_uuid: "0b9a43cf64e54f909cfe47c7f4834ee8" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 36523 } }
I20240502 08:04:12.743052 11075 raft_consensus.cc:729] T 3c700781a5304f31a7ede719eea1ae9a P 9e8f879cc8c44deba93997cc70ce375b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e8f879cc8c44deba93997cc70ce375b, State: Initialized, Role: FOLLOWER
I20240502 08:04:12.743149 11074 raft_consensus.cc:374] T 3c700781a5304f31a7ede719eea1ae9a P 3378106d9722470cb179d01d8e5c9160 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:12.743568 11074 raft_consensus.cc:729] T 3c700781a5304f31a7ede719eea1ae9a P 3378106d9722470cb179d01d8e5c9160 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3378106d9722470cb179d01d8e5c9160, State: Initialized, Role: FOLLOWER
I20240502 08:04:12.743899 11075 consensus_queue.cc:260] T 3c700781a5304f31a7ede719eea1ae9a P 9e8f879cc8c44deba93997cc70ce375b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3378106d9722470cb179d01d8e5c9160" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44223 } } peers { permanent_uuid: "9e8f879cc8c44deba93997cc70ce375b" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 46505 } } peers { permanent_uuid: "0b9a43cf64e54f909cfe47c7f4834ee8" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 36523 } }
I20240502 08:04:12.744272 11074 consensus_queue.cc:260] T 3c700781a5304f31a7ede719eea1ae9a P 3378106d9722470cb179d01d8e5c9160 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3378106d9722470cb179d01d8e5c9160" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44223 } } peers { permanent_uuid: "9e8f879cc8c44deba93997cc70ce375b" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 46505 } } peers { permanent_uuid: "0b9a43cf64e54f909cfe47c7f4834ee8" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 36523 } }
I20240502 08:04:12.745285 11076 raft_consensus.cc:348] T 3c700781a5304f31a7ede719eea1ae9a P 0b9a43cf64e54f909cfe47c7f4834ee8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3378106d9722470cb179d01d8e5c9160" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44223 } } peers { permanent_uuid: "9e8f879cc8c44deba93997cc70ce375b" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 46505 } } peers { permanent_uuid: "0b9a43cf64e54f909cfe47c7f4834ee8" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 36523 } }
I20240502 08:04:12.746053 11076 raft_consensus.cc:374] T 3c700781a5304f31a7ede719eea1ae9a P 0b9a43cf64e54f909cfe47c7f4834ee8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:12.746385 11076 raft_consensus.cc:729] T 3c700781a5304f31a7ede719eea1ae9a P 0b9a43cf64e54f909cfe47c7f4834ee8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b9a43cf64e54f909cfe47c7f4834ee8, State: Initialized, Role: FOLLOWER
I20240502 08:04:12.747185 11076 consensus_queue.cc:260] T 3c700781a5304f31a7ede719eea1ae9a P 0b9a43cf64e54f909cfe47c7f4834ee8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3378106d9722470cb179d01d8e5c9160" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44223 } } peers { permanent_uuid: "9e8f879cc8c44deba93997cc70ce375b" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 46505 } } peers { permanent_uuid: "0b9a43cf64e54f909cfe47c7f4834ee8" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 36523 } }
I20240502 08:04:12.749681 11075 ts_tablet_manager.cc:1398] T 3c700781a5304f31a7ede719eea1ae9a P 9e8f879cc8c44deba93997cc70ce375b: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20240502 08:04:12.750319 11074 ts_tablet_manager.cc:1398] T 3c700781a5304f31a7ede719eea1ae9a P 3378106d9722470cb179d01d8e5c9160: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20240502 08:04:12.753571 11076 ts_tablet_manager.cc:1398] T 3c700781a5304f31a7ede719eea1ae9a P 0b9a43cf64e54f909cfe47c7f4834ee8: Time spent starting tablet: real 0.011s user 0.009s sys 0.000s
W20240502 08:04:12.833307 11055 tablet.cc:2369] T 3c700781a5304f31a7ede719eea1ae9a P 0b9a43cf64e54f909cfe47c7f4834ee8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 08:04:13.000111 11080 raft_consensus.cc:482] T 3c700781a5304f31a7ede719eea1ae9a P 9e8f879cc8c44deba93997cc70ce375b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:04:13.000605 11080 raft_consensus.cc:504] T 3c700781a5304f31a7ede719eea1ae9a P 9e8f879cc8c44deba93997cc70ce375b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3378106d9722470cb179d01d8e5c9160" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44223 } } peers { permanent_uuid: "9e8f879cc8c44deba93997cc70ce375b" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 46505 } } peers { permanent_uuid: "0b9a43cf64e54f909cfe47c7f4834ee8" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 36523 } }
I20240502 08:04:13.001320 11081 raft_consensus.cc:482] T 3c700781a5304f31a7ede719eea1ae9a P 3378106d9722470cb179d01d8e5c9160 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:04:13.001773 11081 raft_consensus.cc:504] T 3c700781a5304f31a7ede719eea1ae9a P 3378106d9722470cb179d01d8e5c9160 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3378106d9722470cb179d01d8e5c9160" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44223 } } peers { permanent_uuid: "9e8f879cc8c44deba93997cc70ce375b" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 46505 } } peers { permanent_uuid: "0b9a43cf64e54f909cfe47c7f4834ee8" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 36523 } }
I20240502 08:04:13.004860 11080 leader_election.cc:288] T 3c700781a5304f31a7ede719eea1ae9a P 9e8f879cc8c44deba93997cc70ce375b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3378106d9722470cb179d01d8e5c9160 (127.6.204.194:44223), 0b9a43cf64e54f909cfe47c7f4834ee8 (127.6.204.195:36523)
I20240502 08:04:13.005887 11081 leader_election.cc:288] T 3c700781a5304f31a7ede719eea1ae9a P 3378106d9722470cb179d01d8e5c9160 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9e8f879cc8c44deba93997cc70ce375b (127.6.204.193:46505), 0b9a43cf64e54f909cfe47c7f4834ee8 (127.6.204.195:36523)
I20240502 08:04:13.020171 10955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c700781a5304f31a7ede719eea1ae9a" candidate_uuid: "9e8f879cc8c44deba93997cc70ce375b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3378106d9722470cb179d01d8e5c9160" is_pre_election: true
I20240502 08:04:13.021153 10955 raft_consensus.cc:2454] T 3c700781a5304f31a7ede719eea1ae9a P 3378106d9722470cb179d01d8e5c9160 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e8f879cc8c44deba93997cc70ce375b in term 0.
I20240502 08:04:13.022950 10880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c700781a5304f31a7ede719eea1ae9a" candidate_uuid: "3378106d9722470cb179d01d8e5c9160" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e8f879cc8c44deba93997cc70ce375b" is_pre_election: true
I20240502 08:04:13.023468 10843 leader_election.cc:302] T 3c700781a5304f31a7ede719eea1ae9a P 9e8f879cc8c44deba93997cc70ce375b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3378106d9722470cb179d01d8e5c9160, 9e8f879cc8c44deba93997cc70ce375b; no voters:
I20240502 08:04:13.023706 10880 raft_consensus.cc:2454] T 3c700781a5304f31a7ede719eea1ae9a P 9e8f879cc8c44deba93997cc70ce375b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3378106d9722470cb179d01d8e5c9160 in term 0.
I20240502 08:04:13.025141 10919 leader_election.cc:302] T 3c700781a5304f31a7ede719eea1ae9a P 3378106d9722470cb179d01d8e5c9160 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3378106d9722470cb179d01d8e5c9160, 9e8f879cc8c44deba93997cc70ce375b; no voters:
I20240502 08:04:13.025156 11080 raft_consensus.cc:2781] T 3c700781a5304f31a7ede719eea1ae9a P 9e8f879cc8c44deba93997cc70ce375b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:04:13.026167 11081 raft_consensus.cc:2781] T 3c700781a5304f31a7ede719eea1ae9a P 3378106d9722470cb179d01d8e5c9160 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:04:13.026315 11080 raft_consensus.cc:482] T 3c700781a5304f31a7ede719eea1ae9a P 9e8f879cc8c44deba93997cc70ce375b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:04:13.027552 11081 raft_consensus.cc:482] T 3c700781a5304f31a7ede719eea1ae9a P 3378106d9722470cb179d01d8e5c9160 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:04:13.027061 11029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c700781a5304f31a7ede719eea1ae9a" candidate_uuid: "9e8f879cc8c44deba93997cc70ce375b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b9a43cf64e54f909cfe47c7f4834ee8" is_pre_election: true
I20240502 08:04:13.027735 11080 raft_consensus.cc:3037] T 3c700781a5304f31a7ede719eea1ae9a P 9e8f879cc8c44deba93997cc70ce375b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:13.028049 11081 raft_consensus.cc:3037] T 3c700781a5304f31a7ede719eea1ae9a P 3378106d9722470cb179d01d8e5c9160 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:13.028259 11029 raft_consensus.cc:2454] T 3c700781a5304f31a7ede719eea1ae9a P 0b9a43cf64e54f909cfe47c7f4834ee8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e8f879cc8c44deba93997cc70ce375b in term 0.
I20240502 08:04:13.033116 11029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c700781a5304f31a7ede719eea1ae9a" candidate_uuid: "3378106d9722470cb179d01d8e5c9160" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b9a43cf64e54f909cfe47c7f4834ee8" is_pre_election: true
I20240502 08:04:13.033690 11029 raft_consensus.cc:2454] T 3c700781a5304f31a7ede719eea1ae9a P 0b9a43cf64e54f909cfe47c7f4834ee8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3378106d9722470cb179d01d8e5c9160 in term 0.
I20240502 08:04:13.035535 11080 raft_consensus.cc:504] T 3c700781a5304f31a7ede719eea1ae9a P 9e8f879cc8c44deba93997cc70ce375b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3378106d9722470cb179d01d8e5c9160" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44223 } } peers { permanent_uuid: "9e8f879cc8c44deba93997cc70ce375b" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 46505 } } peers { permanent_uuid: "0b9a43cf64e54f909cfe47c7f4834ee8" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 36523 } }
I20240502 08:04:13.035535 11081 raft_consensus.cc:504] T 3c700781a5304f31a7ede719eea1ae9a P 3378106d9722470cb179d01d8e5c9160 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3378106d9722470cb179d01d8e5c9160" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44223 } } peers { permanent_uuid: "9e8f879cc8c44deba93997cc70ce375b" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 46505 } } peers { permanent_uuid: "0b9a43cf64e54f909cfe47c7f4834ee8" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 36523 } }
I20240502 08:04:13.037742 11080 leader_election.cc:288] T 3c700781a5304f31a7ede719eea1ae9a P 9e8f879cc8c44deba93997cc70ce375b [CANDIDATE]: Term 1 election: Requested vote from peers 3378106d9722470cb179d01d8e5c9160 (127.6.204.194:44223), 0b9a43cf64e54f909cfe47c7f4834ee8 (127.6.204.195:36523)
I20240502 08:04:13.038852 10880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c700781a5304f31a7ede719eea1ae9a" candidate_uuid: "3378106d9722470cb179d01d8e5c9160" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e8f879cc8c44deba93997cc70ce375b"
I20240502 08:04:13.038853 11029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c700781a5304f31a7ede719eea1ae9a" candidate_uuid: "9e8f879cc8c44deba93997cc70ce375b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b9a43cf64e54f909cfe47c7f4834ee8"
I20240502 08:04:13.039418 11028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c700781a5304f31a7ede719eea1ae9a" candidate_uuid: "3378106d9722470cb179d01d8e5c9160" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b9a43cf64e54f909cfe47c7f4834ee8"
I20240502 08:04:13.039721 10880 raft_consensus.cc:2379] T 3c700781a5304f31a7ede719eea1ae9a P 9e8f879cc8c44deba93997cc70ce375b [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 3378106d9722470cb179d01d8e5c9160 in current term 1: Already voted for candidate 9e8f879cc8c44deba93997cc70ce375b in this term.
I20240502 08:04:13.038933 10955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c700781a5304f31a7ede719eea1ae9a" candidate_uuid: "9e8f879cc8c44deba93997cc70ce375b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3378106d9722470cb179d01d8e5c9160"
I20240502 08:04:13.039711 11029 raft_consensus.cc:3037] T 3c700781a5304f31a7ede719eea1ae9a P 0b9a43cf64e54f909cfe47c7f4834ee8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:13.040803 10955 raft_consensus.cc:2379] T 3c700781a5304f31a7ede719eea1ae9a P 3378106d9722470cb179d01d8e5c9160 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 9e8f879cc8c44deba93997cc70ce375b in current term 1: Already voted for candidate 3378106d9722470cb179d01d8e5c9160 in this term.
I20240502 08:04:13.038925 11081 leader_election.cc:288] T 3c700781a5304f31a7ede719eea1ae9a P 3378106d9722470cb179d01d8e5c9160 [CANDIDATE]: Term 1 election: Requested vote from peers 9e8f879cc8c44deba93997cc70ce375b (127.6.204.193:46505), 0b9a43cf64e54f909cfe47c7f4834ee8 (127.6.204.195:36523)
I20240502 08:04:13.041666 10921 leader_election.cc:302] T 3c700781a5304f31a7ede719eea1ae9a P 3378106d9722470cb179d01d8e5c9160 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3378106d9722470cb179d01d8e5c9160; no voters: 0b9a43cf64e54f909cfe47c7f4834ee8, 9e8f879cc8c44deba93997cc70ce375b
I20240502 08:04:13.042552 11081 raft_consensus.cc:2726] T 3c700781a5304f31a7ede719eea1ae9a P 3378106d9722470cb179d01d8e5c9160 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 08:04:13.045950 11029 raft_consensus.cc:2454] T 3c700781a5304f31a7ede719eea1ae9a P 0b9a43cf64e54f909cfe47c7f4834ee8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e8f879cc8c44deba93997cc70ce375b in term 1.
I20240502 08:04:13.046917 10846 leader_election.cc:302] T 3c700781a5304f31a7ede719eea1ae9a P 9e8f879cc8c44deba93997cc70ce375b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b9a43cf64e54f909cfe47c7f4834ee8, 9e8f879cc8c44deba93997cc70ce375b; no voters: 3378106d9722470cb179d01d8e5c9160
I20240502 08:04:13.047545 11080 raft_consensus.cc:2781] T 3c700781a5304f31a7ede719eea1ae9a P 9e8f879cc8c44deba93997cc70ce375b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:04:13.048588 11080 raft_consensus.cc:686] T 3c700781a5304f31a7ede719eea1ae9a P 9e8f879cc8c44deba93997cc70ce375b [term 1 LEADER]: Becoming Leader. State: Replica: 9e8f879cc8c44deba93997cc70ce375b, State: Running, Role: LEADER
I20240502 08:04:13.049229 11080 consensus_queue.cc:237] T 3c700781a5304f31a7ede719eea1ae9a P 9e8f879cc8c44deba93997cc70ce375b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3378106d9722470cb179d01d8e5c9160" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44223 } } peers { permanent_uuid: "9e8f879cc8c44deba93997cc70ce375b" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 46505 } } peers { permanent_uuid: "0b9a43cf64e54f909cfe47c7f4834ee8" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 36523 } }
I20240502 08:04:13.057225 10776 catalog_manager.cc:5484] T 3c700781a5304f31a7ede719eea1ae9a P 9e8f879cc8c44deba93997cc70ce375b reported cstate change: term changed from 0 to 1, leader changed from <none> to 9e8f879cc8c44deba93997cc70ce375b (127.6.204.193). New cstate: current_term: 1 leader_uuid: "9e8f879cc8c44deba93997cc70ce375b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3378106d9722470cb179d01d8e5c9160" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e8f879cc8c44deba93997cc70ce375b" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 46505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b9a43cf64e54f909cfe47c7f4834ee8" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 36523 } health_report { overall_health: UNKNOWN } } }
I20240502 08:04:13.072834 6963 tablet_server.cc:178] TabletServer@127.6.204.193:0 shutting down...
I20240502 08:04:13.088941 6963 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 08:04:13.089701 6963 tablet_replica.cc:330] T 3c700781a5304f31a7ede719eea1ae9a P 9e8f879cc8c44deba93997cc70ce375b: stopping tablet replica
I20240502 08:04:13.090265 6963 raft_consensus.cc:2229] T 3c700781a5304f31a7ede719eea1ae9a P 9e8f879cc8c44deba93997cc70ce375b [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:04:13.090821 6963 pending_rounds.cc:62] T 3c700781a5304f31a7ede719eea1ae9a P 9e8f879cc8c44deba93997cc70ce375b: Trying to abort 1 pending ops.
I20240502 08:04:13.091027 6963 pending_rounds.cc:68] T 3c700781a5304f31a7ede719eea1ae9a P 9e8f879cc8c44deba93997cc70ce375b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023153369296424960 op_type: NO_OP noop_request { }
I20240502 08:04:13.091348 6963 raft_consensus.cc:2866] T 3c700781a5304f31a7ede719eea1ae9a P 9e8f879cc8c44deba93997cc70ce375b [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240502 08:04:13.091574 6963 raft_consensus.cc:2258] T 3c700781a5304f31a7ede719eea1ae9a P 9e8f879cc8c44deba93997cc70ce375b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:04:13.111335 6963 tablet_server.cc:194] TabletServer@127.6.204.193:0 shutdown complete.
I20240502 08:04:13.202601 10831 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:04:13.203927 10831 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:04:13.204237 10831 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:04:15.126394 11091 raft_consensus.cc:482] T 3c700781a5304f31a7ede719eea1ae9a P 3378106d9722470cb179d01d8e5c9160 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:04:15.127277 11091 raft_consensus.cc:504] T 3c700781a5304f31a7ede719eea1ae9a P 3378106d9722470cb179d01d8e5c9160 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3378106d9722470cb179d01d8e5c9160" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44223 } } peers { permanent_uuid: "9e8f879cc8c44deba93997cc70ce375b" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 46505 } } peers { permanent_uuid: "0b9a43cf64e54f909cfe47c7f4834ee8" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 36523 } }
I20240502 08:04:15.129565 11091 leader_election.cc:288] T 3c700781a5304f31a7ede719eea1ae9a P 3378106d9722470cb179d01d8e5c9160 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9e8f879cc8c44deba93997cc70ce375b (127.6.204.193:46505), 0b9a43cf64e54f909cfe47c7f4834ee8 (127.6.204.195:36523)
I20240502 08:04:15.131268 11029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c700781a5304f31a7ede719eea1ae9a" candidate_uuid: "3378106d9722470cb179d01d8e5c9160" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b9a43cf64e54f909cfe47c7f4834ee8" is_pre_election: true
I20240502 08:04:15.132215 11029 raft_consensus.cc:2454] T 3c700781a5304f31a7ede719eea1ae9a P 0b9a43cf64e54f909cfe47c7f4834ee8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3378106d9722470cb179d01d8e5c9160 in term 1.
I20240502 08:04:15.133507 10921 leader_election.cc:302] T 3c700781a5304f31a7ede719eea1ae9a P 3378106d9722470cb179d01d8e5c9160 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b9a43cf64e54f909cfe47c7f4834ee8, 3378106d9722470cb179d01d8e5c9160; no voters:
I20240502 08:04:15.134272 11091 raft_consensus.cc:2781] T 3c700781a5304f31a7ede719eea1ae9a P 3378106d9722470cb179d01d8e5c9160 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:04:15.134559 11091 raft_consensus.cc:482] T 3c700781a5304f31a7ede719eea1ae9a P 3378106d9722470cb179d01d8e5c9160 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:04:15.134819 11091 raft_consensus.cc:3037] T 3c700781a5304f31a7ede719eea1ae9a P 3378106d9722470cb179d01d8e5c9160 [term 1 FOLLOWER]: Advancing to term 2
W20240502 08:04:15.135958 10919 leader_election.cc:334] T 3c700781a5304f31a7ede719eea1ae9a P 3378106d9722470cb179d01d8e5c9160 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9e8f879cc8c44deba93997cc70ce375b (127.6.204.193:46505): Network error: Client connection negotiation failed: client connection to 127.6.204.193:46505: connect: Connection refused (error 111)
I20240502 08:04:15.141747 11091 raft_consensus.cc:504] T 3c700781a5304f31a7ede719eea1ae9a P 3378106d9722470cb179d01d8e5c9160 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3378106d9722470cb179d01d8e5c9160" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44223 } } peers { permanent_uuid: "9e8f879cc8c44deba93997cc70ce375b" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 46505 } } peers { permanent_uuid: "0b9a43cf64e54f909cfe47c7f4834ee8" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 36523 } }
I20240502 08:04:15.143299 11091 leader_election.cc:288] T 3c700781a5304f31a7ede719eea1ae9a P 3378106d9722470cb179d01d8e5c9160 [CANDIDATE]: Term 2 election: Requested vote from peers 9e8f879cc8c44deba93997cc70ce375b (127.6.204.193:46505), 0b9a43cf64e54f909cfe47c7f4834ee8 (127.6.204.195:36523)
I20240502 08:04:15.144402 11029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c700781a5304f31a7ede719eea1ae9a" candidate_uuid: "3378106d9722470cb179d01d8e5c9160" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b9a43cf64e54f909cfe47c7f4834ee8"
I20240502 08:04:15.145020 11029 raft_consensus.cc:3037] T 3c700781a5304f31a7ede719eea1ae9a P 0b9a43cf64e54f909cfe47c7f4834ee8 [term 1 FOLLOWER]: Advancing to term 2
W20240502 08:04:15.147002 10919 leader_election.cc:334] T 3c700781a5304f31a7ede719eea1ae9a P 3378106d9722470cb179d01d8e5c9160 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9e8f879cc8c44deba93997cc70ce375b (127.6.204.193:46505): Network error: Client connection negotiation failed: client connection to 127.6.204.193:46505: connect: Connection refused (error 111)
I20240502 08:04:15.151715 11029 raft_consensus.cc:2454] T 3c700781a5304f31a7ede719eea1ae9a P 0b9a43cf64e54f909cfe47c7f4834ee8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3378106d9722470cb179d01d8e5c9160 in term 2.
I20240502 08:04:15.152667 10921 leader_election.cc:302] T 3c700781a5304f31a7ede719eea1ae9a P 3378106d9722470cb179d01d8e5c9160 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b9a43cf64e54f909cfe47c7f4834ee8, 3378106d9722470cb179d01d8e5c9160; no voters: 9e8f879cc8c44deba93997cc70ce375b
I20240502 08:04:15.153299 11091 raft_consensus.cc:2781] T 3c700781a5304f31a7ede719eea1ae9a P 3378106d9722470cb179d01d8e5c9160 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:04:15.154176 11091 raft_consensus.cc:686] T 3c700781a5304f31a7ede719eea1ae9a P 3378106d9722470cb179d01d8e5c9160 [term 2 LEADER]: Becoming Leader. State: Replica: 3378106d9722470cb179d01d8e5c9160, State: Running, Role: LEADER
I20240502 08:04:15.155133 11091 consensus_queue.cc:237] T 3c700781a5304f31a7ede719eea1ae9a P 3378106d9722470cb179d01d8e5c9160 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3378106d9722470cb179d01d8e5c9160" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44223 } } peers { permanent_uuid: "9e8f879cc8c44deba93997cc70ce375b" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 46505 } } peers { permanent_uuid: "0b9a43cf64e54f909cfe47c7f4834ee8" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 36523 } }
I20240502 08:04:15.162361 10776 catalog_manager.cc:5484] T 3c700781a5304f31a7ede719eea1ae9a P 3378106d9722470cb179d01d8e5c9160 reported cstate change: term changed from 1 to 2, leader changed from 9e8f879cc8c44deba93997cc70ce375b (127.6.204.193) to 3378106d9722470cb179d01d8e5c9160 (127.6.204.194). New cstate: current_term: 2 leader_uuid: "3378106d9722470cb179d01d8e5c9160" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3378106d9722470cb179d01d8e5c9160" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e8f879cc8c44deba93997cc70ce375b" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 46505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b9a43cf64e54f909cfe47c7f4834ee8" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 36523 } health_report { overall_health: UNKNOWN } } }
I20240502 08:04:15.205751 10831 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:04:15.207614 10831 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:04:15.208065 10831 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
W20240502 08:04:15.694451 10919 consensus_peers.cc:485] T 3c700781a5304f31a7ede719eea1ae9a P 3378106d9722470cb179d01d8e5c9160 -> Peer 9e8f879cc8c44deba93997cc70ce375b (127.6.204.193:46505): Couldn't send request to peer 9e8f879cc8c44deba93997cc70ce375b. Status: Network error: Client connection negotiation failed: client connection to 127.6.204.193:46505: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:04:15.716287 11029 raft_consensus.cc:1261] T 3c700781a5304f31a7ede719eea1ae9a P 0b9a43cf64e54f909cfe47c7f4834ee8 [term 2 FOLLOWER]: Refusing update from remote peer 3378106d9722470cb179d01d8e5c9160: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240502 08:04:15.717583 11100 consensus_queue.cc:1035] T 3c700781a5304f31a7ede719eea1ae9a P 3378106d9722470cb179d01d8e5c9160 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b9a43cf64e54f909cfe47c7f4834ee8" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 36523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:04:16.089447 10919 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.204.193:46505: connect: Connection refused (error 111) [suppressed 3 similar messages]
I20240502 08:04:17.209105 10831 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:04:17.210476 10831 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:04:17.210736 10831 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
W20240502 08:04:18.061486 10919 consensus_peers.cc:485] T 3c700781a5304f31a7ede719eea1ae9a P 3378106d9722470cb179d01d8e5c9160 -> Peer 9e8f879cc8c44deba93997cc70ce375b (127.6.204.193:46505): Couldn't send request to peer 9e8f879cc8c44deba93997cc70ce375b. Status: Network error: Client connection negotiation failed: client connection to 127.6.204.193:46505: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 08:04:19.211529 10831 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:04:19.212711 10831 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:04:19.212966 10831 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
W20240502 08:04:20.632867 10919 consensus_peers.cc:485] T 3c700781a5304f31a7ede719eea1ae9a P 3378106d9722470cb179d01d8e5c9160 -> Peer 9e8f879cc8c44deba93997cc70ce375b (127.6.204.193:46505): Couldn't send request to peer 9e8f879cc8c44deba93997cc70ce375b. Status: Network error: Client connection negotiation failed: client connection to 127.6.204.193:46505: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:04:21.117472 10919 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.204.193:46505: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 08:04:21.213886 10831 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:04:21.215170 10831 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:04:21.215443 10831 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
W20240502 08:04:22.967089 10919 consensus_peers.cc:485] T 3c700781a5304f31a7ede719eea1ae9a P 3378106d9722470cb179d01d8e5c9160 -> Peer 9e8f879cc8c44deba93997cc70ce375b (127.6.204.193:46505): Couldn't send request to peer 9e8f879cc8c44deba93997cc70ce375b. Status: Network error: Client connection negotiation failed: client connection to 127.6.204.193:46505: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 08:04:23.216364 10831 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 08:04:23.217736 10831 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 08:04:23.218043 10831 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 08:04:24.183045 6963 tablet_server.cc:178] TabletServer@127.6.204.194:0 shutting down...
I20240502 08:04:24.210304 6963 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 08:04:24.211019 6963 tablet_replica.cc:330] T 3c700781a5304f31a7ede719eea1ae9a P 3378106d9722470cb179d01d8e5c9160: stopping tablet replica
I20240502 08:04:24.211694 6963 raft_consensus.cc:2229] T 3c700781a5304f31a7ede719eea1ae9a P 3378106d9722470cb179d01d8e5c9160 [term 2 LEADER]: Raft consensus shutting down.
I20240502 08:04:24.212451 6963 raft_consensus.cc:2258] T 3c700781a5304f31a7ede719eea1ae9a P 3378106d9722470cb179d01d8e5c9160 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:04:24.231801 6963 tablet_server.cc:194] TabletServer@127.6.204.194:0 shutdown complete.
I20240502 08:04:24.244684 6963 tablet_server.cc:178] TabletServer@127.6.204.195:0 shutting down...
I20240502 08:04:24.275336 6963 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 08:04:24.276248 6963 tablet_replica.cc:330] T 3c700781a5304f31a7ede719eea1ae9a P 0b9a43cf64e54f909cfe47c7f4834ee8: stopping tablet replica
I20240502 08:04:24.276911 6963 raft_consensus.cc:2229] T 3c700781a5304f31a7ede719eea1ae9a P 0b9a43cf64e54f909cfe47c7f4834ee8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:04:24.277580 6963 raft_consensus.cc:2258] T 3c700781a5304f31a7ede719eea1ae9a P 0b9a43cf64e54f909cfe47c7f4834ee8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:04:24.298211 6963 tablet_server.cc:194] TabletServer@127.6.204.195:0 shutdown complete.
I20240502 08:04:24.310508 6963 master.cc:537] Master@127.6.204.254:43795 shutting down...
I20240502 08:04:24.334086 6963 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 68987d3adb6f4490ae2f655c7a141fcd [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:04:24.334933 6963 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 68987d3adb6f4490ae2f655c7a141fcd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:04:24.335275 6963 tablet_replica.cc:330] T 00000000000000000000000000000000 P 68987d3adb6f4490ae2f655c7a141fcd: stopping tablet replica
I20240502 08:04:24.353886 6963 master.cc:552] Master@127.6.204.254:43795 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor (13329 ms)
[ RUN ] AutoRebalancerTest.NoRebalancingIfReplicasRecovering
I20240502 08:04:24.391521 6963 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.6.204.254:37875
I20240502 08:04:24.392645 6963 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:04:24.398367 11122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:04:24.398610 11123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:04:24.399291 11125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:04:24.400473 6963 server_base.cc:1034] running on GCE node
I20240502 08:04:24.401319 6963 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:04:24.401504 6963 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:04:24.401654 6963 hybrid_clock.cc:648] HybridClock initialized: now 1714637064401636 us; error 0 us; skew 500 ppm
I20240502 08:04:24.402160 6963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:04:24.404532 6963 webserver.cc:460] Webserver started at http://127.6.204.254:41007/ using document root <none> and password file <none>
I20240502 08:04:24.404990 6963 fs_manager.cc:359] Metadata directory not provided
I20240502 08:04:24.405155 6963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:04:24.405431 6963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:04:24.406754 6963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714636933625983-6963-0/minicluster-data/master-0-root/instance:
uuid: "c6e176cf9790442c95418718e7c71e08"
format_stamp: "Formatted at 2024-05-02 08:04:24 on dist-test-slave-32xs"
I20240502 08:04:24.411151 6963 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240502 08:04:24.414417 11130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:04:24.415089 6963 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 08:04:24.415347 6963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714636933625983-6963-0/minicluster-data/master-0-root
uuid: "c6e176cf9790442c95418718e7c71e08"
format_stamp: "Formatted at 2024-05-02 08:04:24 on dist-test-slave-32xs"
I20240502 08:04:24.415621 6963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714636933625983-6963-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714636933625983-6963-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714636933625983-6963-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:04:24.427750 6963 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:04:24.428890 6963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:04:24.466363 6963 rpc_server.cc:307] RPC server started. Bound to: 127.6.204.254:37875
I20240502 08:04:24.466464 11181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.204.254:37875 every 8 connection(s)
I20240502 08:04:24.470120 11182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:24.481406 11182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6e176cf9790442c95418718e7c71e08: Bootstrap starting.
I20240502 08:04:24.485580 11182 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6e176cf9790442c95418718e7c71e08: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:24.489596 11182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6e176cf9790442c95418718e7c71e08: No bootstrap required, opened a new log
I20240502 08:04:24.491607 11182 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6e176cf9790442c95418718e7c71e08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6e176cf9790442c95418718e7c71e08" member_type: VOTER }
I20240502 08:04:24.492061 11182 raft_consensus.cc:374] T 00000000000000000000000000000000 P c6e176cf9790442c95418718e7c71e08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:24.492246 11182 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6e176cf9790442c95418718e7c71e08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6e176cf9790442c95418718e7c71e08, State: Initialized, Role: FOLLOWER
I20240502 08:04:24.492766 11182 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6e176cf9790442c95418718e7c71e08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6e176cf9790442c95418718e7c71e08" member_type: VOTER }
I20240502 08:04:24.493207 11182 raft_consensus.cc:388] T 00000000000000000000000000000000 P c6e176cf9790442c95418718e7c71e08 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:04:24.493392 11182 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6e176cf9790442c95418718e7c71e08 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:04:24.493619 11182 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6e176cf9790442c95418718e7c71e08 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:24.497977 11182 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6e176cf9790442c95418718e7c71e08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6e176cf9790442c95418718e7c71e08" member_type: VOTER }
I20240502 08:04:24.498489 11182 leader_election.cc:302] T 00000000000000000000000000000000 P c6e176cf9790442c95418718e7c71e08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6e176cf9790442c95418718e7c71e08; no voters:
I20240502 08:04:24.499575 11182 leader_election.cc:288] T 00000000000000000000000000000000 P c6e176cf9790442c95418718e7c71e08 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:04:24.500015 11185 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6e176cf9790442c95418718e7c71e08 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:04:24.501330 11185 raft_consensus.cc:686] T 00000000000000000000000000000000 P c6e176cf9790442c95418718e7c71e08 [term 1 LEADER]: Becoming Leader. State: Replica: c6e176cf9790442c95418718e7c71e08, State: Running, Role: LEADER
I20240502 08:04:24.502058 11185 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6e176cf9790442c95418718e7c71e08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6e176cf9790442c95418718e7c71e08" member_type: VOTER }
I20240502 08:04:24.502583 11182 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6e176cf9790442c95418718e7c71e08 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 08:04:24.505007 11186 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6e176cf9790442c95418718e7c71e08 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c6e176cf9790442c95418718e7c71e08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6e176cf9790442c95418718e7c71e08" member_type: VOTER } }
I20240502 08:04:24.505033 11187 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6e176cf9790442c95418718e7c71e08 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6e176cf9790442c95418718e7c71e08. Latest consensus state: current_term: 1 leader_uuid: "c6e176cf9790442c95418718e7c71e08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6e176cf9790442c95418718e7c71e08" member_type: VOTER } }
I20240502 08:04:24.505872 11187 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6e176cf9790442c95418718e7c71e08 [sys.catalog]: This master's current role is: LEADER
I20240502 08:04:24.505856 11186 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6e176cf9790442c95418718e7c71e08 [sys.catalog]: This master's current role is: LEADER
I20240502 08:04:24.509943 11190 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 08:04:24.518169 11190 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 08:04:24.519243 6963 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 08:04:24.526981 11190 catalog_manager.cc:1347] Generated new cluster ID: 24ace7612bb84ee78eec560eea34b3be
I20240502 08:04:24.527274 11190 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 08:04:24.555262 11190 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 08:04:24.556563 11190 catalog_manager.cc:1504] Loading token signing keys...
I20240502 08:04:24.576372 11190 catalog_manager.cc:5857] T 00000000000000000000000000000000 P c6e176cf9790442c95418718e7c71e08: Generated new TSK 0
I20240502 08:04:24.576980 11190 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 08:04:24.586306 6963 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:04:24.592058 11203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:04:24.593477 11204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:04:24.594735 11206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:04:24.595818 6963 server_base.cc:1034] running on GCE node
I20240502 08:04:24.596571 6963 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:04:24.596752 6963 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:04:24.596899 6963 hybrid_clock.cc:648] HybridClock initialized: now 1714637064596885 us; error 0 us; skew 500 ppm
I20240502 08:04:24.597441 6963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:04:24.599767 6963 webserver.cc:460] Webserver started at http://127.6.204.193:32823/ using document root <none> and password file <none>
I20240502 08:04:24.600214 6963 fs_manager.cc:359] Metadata directory not provided
I20240502 08:04:24.600395 6963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:04:24.600634 6963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:04:24.601907 6963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714636933625983-6963-0/minicluster-data/ts-0-root/instance:
uuid: "c5e60e7e8697400c8c599205f6db37cf"
format_stamp: "Formatted at 2024-05-02 08:04:24 on dist-test-slave-32xs"
I20240502 08:04:24.606406 6963 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20240502 08:04:24.609459 11211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:04:24.610195 6963 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 08:04:24.610477 6963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714636933625983-6963-0/minicluster-data/ts-0-root
uuid: "c5e60e7e8697400c8c599205f6db37cf"
format_stamp: "Formatted at 2024-05-02 08:04:24 on dist-test-slave-32xs"
I20240502 08:04:24.610750 6963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714636933625983-6963-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714636933625983-6963-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714636933625983-6963-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:04:24.623406 6963 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:04:24.624552 6963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:04:24.625986 6963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:04:24.628240 6963 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:04:24.628437 6963 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:04:24.628662 6963 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:04:24.628834 6963 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:04:24.668460 6963 rpc_server.cc:307] RPC server started. Bound to: 127.6.204.193:39769
I20240502 08:04:24.668555 11273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.204.193:39769 every 8 connection(s)
I20240502 08:04:24.673167 6963 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:04:24.680851 11278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:04:24.683596 11279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:04:24.686582 11281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:04:24.687155 11274 heartbeater.cc:347] Connected to a master server at 127.6.204.254:37875
I20240502 08:04:24.687503 11274 heartbeater.cc:464] Registering TS with master...
I20240502 08:04:24.687719 6963 server_base.cc:1034] running on GCE node
I20240502 08:04:24.688469 11274 heartbeater.cc:511] Master 127.6.204.254:37875 requested a full tablet report, sending...
I20240502 08:04:24.688884 6963 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:04:24.689147 6963 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:04:24.689348 6963 hybrid_clock.cc:648] HybridClock initialized: now 1714637064689327 us; error 0 us; skew 500 ppm
I20240502 08:04:24.690096 6963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:04:24.690774 11147 ts_manager.cc:194] Registered new tserver with Master: c5e60e7e8697400c8c599205f6db37cf (127.6.204.193:39769)
I20240502 08:04:24.692824 11147 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52796
I20240502 08:04:24.692898 6963 webserver.cc:460] Webserver started at http://127.6.204.194:41515/ using document root <none> and password file <none>
I20240502 08:04:24.693423 6963 fs_manager.cc:359] Metadata directory not provided
I20240502 08:04:24.693601 6963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:04:24.693866 6963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:04:24.695482 6963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714636933625983-6963-0/minicluster-data/ts-1-root/instance:
uuid: "122c848014b445728238d1ba49ac509c"
format_stamp: "Formatted at 2024-05-02 08:04:24 on dist-test-slave-32xs"
I20240502 08:04:24.700217 6963 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20240502 08:04:24.703562 11286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:04:24.704380 6963 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 08:04:24.704669 6963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714636933625983-6963-0/minicluster-data/ts-1-root
uuid: "122c848014b445728238d1ba49ac509c"
format_stamp: "Formatted at 2024-05-02 08:04:24 on dist-test-slave-32xs"
I20240502 08:04:24.704952 6963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714636933625983-6963-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714636933625983-6963-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714636933625983-6963-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:04:24.724066 6963 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:04:24.725265 6963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:04:24.726680 6963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:04:24.728840 6963 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:04:24.729038 6963 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:04:24.729254 6963 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:04:24.729404 6963 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:04:24.768110 6963 rpc_server.cc:307] RPC server started. Bound to: 127.6.204.194:44945
I20240502 08:04:24.768226 11348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.204.194:44945 every 8 connection(s)
I20240502 08:04:24.772833 6963 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:04:24.779220 11352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:04:24.779953 11353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:04:24.784838 11355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:04:24.785917 11349 heartbeater.cc:347] Connected to a master server at 127.6.204.254:37875
I20240502 08:04:24.786329 11349 heartbeater.cc:464] Registering TS with master...
I20240502 08:04:24.786664 6963 server_base.cc:1034] running on GCE node
I20240502 08:04:24.787186 11349 heartbeater.cc:511] Master 127.6.204.254:37875 requested a full tablet report, sending...
I20240502 08:04:24.787554 6963 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:04:24.787768 6963 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:04:24.787942 6963 hybrid_clock.cc:648] HybridClock initialized: now 1714637064787921 us; error 0 us; skew 500 ppm
I20240502 08:04:24.788563 6963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:04:24.789216 11147 ts_manager.cc:194] Registered new tserver with Master: 122c848014b445728238d1ba49ac509c (127.6.204.194:44945)
I20240502 08:04:24.791182 11147 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52800
I20240502 08:04:24.791568 6963 webserver.cc:460] Webserver started at http://127.6.204.195:46171/ using document root <none> and password file <none>
I20240502 08:04:24.792178 6963 fs_manager.cc:359] Metadata directory not provided
I20240502 08:04:24.792390 6963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:04:24.792646 6963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:04:24.793812 6963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714636933625983-6963-0/minicluster-data/ts-2-root/instance:
uuid: "f13c7769fc9f4bfa9796a5c29f715fb4"
format_stamp: "Formatted at 2024-05-02 08:04:24 on dist-test-slave-32xs"
I20240502 08:04:24.798245 6963 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20240502 08:04:24.801628 11360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:04:24.802562 6963 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 08:04:24.802855 6963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714636933625983-6963-0/minicluster-data/ts-2-root
uuid: "f13c7769fc9f4bfa9796a5c29f715fb4"
format_stamp: "Formatted at 2024-05-02 08:04:24 on dist-test-slave-32xs"
I20240502 08:04:24.803122 6963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714636933625983-6963-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714636933625983-6963-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714636933625983-6963-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:04:24.819749 6963 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:04:24.820865 6963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:04:24.822355 6963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:04:24.824568 6963 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:04:24.824752 6963 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:04:24.825002 6963 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:04:24.825145 6963 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:04:24.864784 6963 rpc_server.cc:307] RPC server started. Bound to: 127.6.204.195:46495
I20240502 08:04:24.864881 11422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.204.195:46495 every 8 connection(s)
I20240502 08:04:24.878806 11423 heartbeater.cc:347] Connected to a master server at 127.6.204.254:37875
I20240502 08:04:24.879144 11423 heartbeater.cc:464] Registering TS with master...
I20240502 08:04:24.879844 11423 heartbeater.cc:511] Master 127.6.204.254:37875 requested a full tablet report, sending...
I20240502 08:04:24.881671 11147 ts_manager.cc:194] Registered new tserver with Master: f13c7769fc9f4bfa9796a5c29f715fb4 (127.6.204.195:46495)
I20240502 08:04:24.882218 6963 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.014420506s
I20240502 08:04:24.883471 11147 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52802
I20240502 08:04:25.695746 11274 heartbeater.cc:503] Master 127.6.204.254:37875 was elected leader, sending a full tablet report...
I20240502 08:04:25.793828 11349 heartbeater.cc:503] Master 127.6.204.254:37875 was elected leader, sending a full tablet report...
I20240502 08:04:25.886209 11423 heartbeater.cc:503] Master 127.6.204.254:37875 was elected leader, sending a full tablet report...
I20240502 08:04:25.915484 6963 test_util.cc:268] Using random seed: 222006332
I20240502 08:04:25.937793 11147 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52810:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:04:25.940126 11147 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:04:25.991065 11314 tablet_service.cc:1467] Processing CreateTablet for tablet d117f9452626438cb18dda39df6eb6ed (DEFAULT_TABLE table=test-workload [id=074ab7bcef0f4688939a5810758e98a7]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240502 08:04:25.991307 11313 tablet_service.cc:1467] Processing CreateTablet for tablet 8c2569709c52401cae6702e5e931ca83 (DEFAULT_TABLE table=test-workload [id=074ab7bcef0f4688939a5810758e98a7]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240502 08:04:25.992660 11314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d117f9452626438cb18dda39df6eb6ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:25.992488 11311 tablet_service.cc:1467] Processing CreateTablet for tablet 743666c9f9a144978e534923cc63222e (DEFAULT_TABLE table=test-workload [id=074ab7bcef0f4688939a5810758e98a7]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240502 08:04:25.993336 11313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c2569709c52401cae6702e5e931ca83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:25.994725 11312 tablet_service.cc:1467] Processing CreateTablet for tablet f3666c62e67047bb9393d96859b539eb (DEFAULT_TABLE table=test-workload [id=074ab7bcef0f4688939a5810758e98a7]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240502 08:04:25.996098 11312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3666c62e67047bb9393d96859b539eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:26.003001 11311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 743666c9f9a144978e534923cc63222e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:26.004671 11239 tablet_service.cc:1467] Processing CreateTablet for tablet d117f9452626438cb18dda39df6eb6ed (DEFAULT_TABLE table=test-workload [id=074ab7bcef0f4688939a5810758e98a7]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240502 08:04:26.006019 11239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d117f9452626438cb18dda39df6eb6ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:26.012290 11238 tablet_service.cc:1467] Processing CreateTablet for tablet 8c2569709c52401cae6702e5e931ca83 (DEFAULT_TABLE table=test-workload [id=074ab7bcef0f4688939a5810758e98a7]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240502 08:04:26.012676 11388 tablet_service.cc:1467] Processing CreateTablet for tablet d117f9452626438cb18dda39df6eb6ed (DEFAULT_TABLE table=test-workload [id=074ab7bcef0f4688939a5810758e98a7]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240502 08:04:26.013713 11238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c2569709c52401cae6702e5e931ca83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:26.013842 11388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d117f9452626438cb18dda39df6eb6ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:26.017012 11237 tablet_service.cc:1467] Processing CreateTablet for tablet f3666c62e67047bb9393d96859b539eb (DEFAULT_TABLE table=test-workload [id=074ab7bcef0f4688939a5810758e98a7]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240502 08:04:26.017784 11387 tablet_service.cc:1467] Processing CreateTablet for tablet 8c2569709c52401cae6702e5e931ca83 (DEFAULT_TABLE table=test-workload [id=074ab7bcef0f4688939a5810758e98a7]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240502 08:04:26.018415 11237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3666c62e67047bb9393d96859b539eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:26.019105 11387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c2569709c52401cae6702e5e931ca83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:26.024205 11236 tablet_service.cc:1467] Processing CreateTablet for tablet 743666c9f9a144978e534923cc63222e (DEFAULT_TABLE table=test-workload [id=074ab7bcef0f4688939a5810758e98a7]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240502 08:04:26.025477 11236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 743666c9f9a144978e534923cc63222e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:26.027865 11386 tablet_service.cc:1467] Processing CreateTablet for tablet f3666c62e67047bb9393d96859b539eb (DEFAULT_TABLE table=test-workload [id=074ab7bcef0f4688939a5810758e98a7]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240502 08:04:26.029177 11386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3666c62e67047bb9393d96859b539eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:26.031409 11385 tablet_service.cc:1467] Processing CreateTablet for tablet 743666c9f9a144978e534923cc63222e (DEFAULT_TABLE table=test-workload [id=074ab7bcef0f4688939a5810758e98a7]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240502 08:04:26.032665 11385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 743666c9f9a144978e534923cc63222e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:26.047401 11443 tablet_bootstrap.cc:492] T d117f9452626438cb18dda39df6eb6ed P 122c848014b445728238d1ba49ac509c: Bootstrap starting.
I20240502 08:04:26.055851 11445 tablet_bootstrap.cc:492] T d117f9452626438cb18dda39df6eb6ed P c5e60e7e8697400c8c599205f6db37cf: Bootstrap starting.
I20240502 08:04:26.057404 11443 tablet_bootstrap.cc:654] T d117f9452626438cb18dda39df6eb6ed P 122c848014b445728238d1ba49ac509c: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:26.063199 11445 tablet_bootstrap.cc:654] T d117f9452626438cb18dda39df6eb6ed P c5e60e7e8697400c8c599205f6db37cf: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:26.060063 11444 tablet_bootstrap.cc:492] T f3666c62e67047bb9393d96859b539eb P f13c7769fc9f4bfa9796a5c29f715fb4: Bootstrap starting.
I20240502 08:04:26.072327 11444 tablet_bootstrap.cc:654] T f3666c62e67047bb9393d96859b539eb P f13c7769fc9f4bfa9796a5c29f715fb4: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:26.080096 11443 tablet_bootstrap.cc:492] T d117f9452626438cb18dda39df6eb6ed P 122c848014b445728238d1ba49ac509c: No bootstrap required, opened a new log
I20240502 08:04:26.081295 11443 ts_tablet_manager.cc:1367] T d117f9452626438cb18dda39df6eb6ed P 122c848014b445728238d1ba49ac509c: Time spent bootstrapping tablet: real 0.034s user 0.013s sys 0.005s
I20240502 08:04:26.083153 11445 tablet_bootstrap.cc:492] T d117f9452626438cb18dda39df6eb6ed P c5e60e7e8697400c8c599205f6db37cf: No bootstrap required, opened a new log
I20240502 08:04:26.083591 11445 ts_tablet_manager.cc:1367] T d117f9452626438cb18dda39df6eb6ed P c5e60e7e8697400c8c599205f6db37cf: Time spent bootstrapping tablet: real 0.028s user 0.015s sys 0.004s
I20240502 08:04:26.083904 11444 tablet_bootstrap.cc:492] T f3666c62e67047bb9393d96859b539eb P f13c7769fc9f4bfa9796a5c29f715fb4: No bootstrap required, opened a new log
I20240502 08:04:26.084406 11444 ts_tablet_manager.cc:1367] T f3666c62e67047bb9393d96859b539eb P f13c7769fc9f4bfa9796a5c29f715fb4: Time spent bootstrapping tablet: real 0.025s user 0.007s sys 0.004s
I20240502 08:04:26.084133 11443 raft_consensus.cc:348] T d117f9452626438cb18dda39df6eb6ed P 122c848014b445728238d1ba49ac509c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } }
I20240502 08:04:26.084909 11443 raft_consensus.cc:374] T d117f9452626438cb18dda39df6eb6ed P 122c848014b445728238d1ba49ac509c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:26.085237 11443 raft_consensus.cc:729] T d117f9452626438cb18dda39df6eb6ed P 122c848014b445728238d1ba49ac509c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 122c848014b445728238d1ba49ac509c, State: Initialized, Role: FOLLOWER
I20240502 08:04:26.085644 11445 raft_consensus.cc:348] T d117f9452626438cb18dda39df6eb6ed P c5e60e7e8697400c8c599205f6db37cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } }
I20240502 08:04:26.086298 11445 raft_consensus.cc:374] T d117f9452626438cb18dda39df6eb6ed P c5e60e7e8697400c8c599205f6db37cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:26.085945 11443 consensus_queue.cc:260] T d117f9452626438cb18dda39df6eb6ed P 122c848014b445728238d1ba49ac509c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } }
I20240502 08:04:26.086540 11445 raft_consensus.cc:729] T d117f9452626438cb18dda39df6eb6ed P c5e60e7e8697400c8c599205f6db37cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e60e7e8697400c8c599205f6db37cf, State: Initialized, Role: FOLLOWER
I20240502 08:04:26.086936 11444 raft_consensus.cc:348] T f3666c62e67047bb9393d96859b539eb P f13c7769fc9f4bfa9796a5c29f715fb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } } peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } }
I20240502 08:04:26.087267 11445 consensus_queue.cc:260] T d117f9452626438cb18dda39df6eb6ed P c5e60e7e8697400c8c599205f6db37cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } }
I20240502 08:04:26.087882 11444 raft_consensus.cc:374] T f3666c62e67047bb9393d96859b539eb P f13c7769fc9f4bfa9796a5c29f715fb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:26.088276 11444 raft_consensus.cc:729] T f3666c62e67047bb9393d96859b539eb P f13c7769fc9f4bfa9796a5c29f715fb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f13c7769fc9f4bfa9796a5c29f715fb4, State: Initialized, Role: FOLLOWER
I20240502 08:04:26.089073 11443 ts_tablet_manager.cc:1398] T d117f9452626438cb18dda39df6eb6ed P 122c848014b445728238d1ba49ac509c: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20240502 08:04:26.090791 11445 ts_tablet_manager.cc:1398] T d117f9452626438cb18dda39df6eb6ed P c5e60e7e8697400c8c599205f6db37cf: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20240502 08:04:26.090952 11443 tablet_bootstrap.cc:492] T f3666c62e67047bb9393d96859b539eb P 122c848014b445728238d1ba49ac509c: Bootstrap starting.
I20240502 08:04:26.089100 11444 consensus_queue.cc:260] T f3666c62e67047bb9393d96859b539eb P f13c7769fc9f4bfa9796a5c29f715fb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } } peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } }
I20240502 08:04:26.091907 11445 tablet_bootstrap.cc:492] T 743666c9f9a144978e534923cc63222e P c5e60e7e8697400c8c599205f6db37cf: Bootstrap starting.
I20240502 08:04:26.098101 11445 tablet_bootstrap.cc:654] T 743666c9f9a144978e534923cc63222e P c5e60e7e8697400c8c599205f6db37cf: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:26.098773 11443 tablet_bootstrap.cc:654] T f3666c62e67047bb9393d96859b539eb P 122c848014b445728238d1ba49ac509c: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:26.123260 11444 ts_tablet_manager.cc:1398] T f3666c62e67047bb9393d96859b539eb P f13c7769fc9f4bfa9796a5c29f715fb4: Time spent starting tablet: real 0.039s user 0.019s sys 0.018s
I20240502 08:04:26.124320 11444 tablet_bootstrap.cc:492] T d117f9452626438cb18dda39df6eb6ed P f13c7769fc9f4bfa9796a5c29f715fb4: Bootstrap starting.
I20240502 08:04:26.124753 11445 tablet_bootstrap.cc:492] T 743666c9f9a144978e534923cc63222e P c5e60e7e8697400c8c599205f6db37cf: No bootstrap required, opened a new log
I20240502 08:04:26.125730 11445 ts_tablet_manager.cc:1367] T 743666c9f9a144978e534923cc63222e P c5e60e7e8697400c8c599205f6db37cf: Time spent bootstrapping tablet: real 0.034s user 0.025s sys 0.008s
I20240502 08:04:26.128779 11444 tablet_bootstrap.cc:654] T d117f9452626438cb18dda39df6eb6ed P f13c7769fc9f4bfa9796a5c29f715fb4: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:26.128469 11445 raft_consensus.cc:348] T 743666c9f9a144978e534923cc63222e P c5e60e7e8697400c8c599205f6db37cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } } peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } }
I20240502 08:04:26.129333 11445 raft_consensus.cc:374] T 743666c9f9a144978e534923cc63222e P c5e60e7e8697400c8c599205f6db37cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:26.129704 11445 raft_consensus.cc:729] T 743666c9f9a144978e534923cc63222e P c5e60e7e8697400c8c599205f6db37cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e60e7e8697400c8c599205f6db37cf, State: Initialized, Role: FOLLOWER
I20240502 08:04:26.130528 11443 tablet_bootstrap.cc:492] T f3666c62e67047bb9393d96859b539eb P 122c848014b445728238d1ba49ac509c: No bootstrap required, opened a new log
I20240502 08:04:26.131135 11443 ts_tablet_manager.cc:1367] T f3666c62e67047bb9393d96859b539eb P 122c848014b445728238d1ba49ac509c: Time spent bootstrapping tablet: real 0.040s user 0.014s sys 0.020s
I20240502 08:04:26.130611 11445 consensus_queue.cc:260] T 743666c9f9a144978e534923cc63222e P c5e60e7e8697400c8c599205f6db37cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } } peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } }
I20240502 08:04:26.133919 11443 raft_consensus.cc:348] T f3666c62e67047bb9393d96859b539eb P 122c848014b445728238d1ba49ac509c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } } peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } }
I20240502 08:04:26.134761 11443 raft_consensus.cc:374] T f3666c62e67047bb9393d96859b539eb P 122c848014b445728238d1ba49ac509c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:26.135107 11443 raft_consensus.cc:729] T f3666c62e67047bb9393d96859b539eb P 122c848014b445728238d1ba49ac509c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 122c848014b445728238d1ba49ac509c, State: Initialized, Role: FOLLOWER
I20240502 08:04:26.136008 11443 consensus_queue.cc:260] T f3666c62e67047bb9393d96859b539eb P 122c848014b445728238d1ba49ac509c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } } peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } }
I20240502 08:04:26.136549 11445 ts_tablet_manager.cc:1398] T 743666c9f9a144978e534923cc63222e P c5e60e7e8697400c8c599205f6db37cf: Time spent starting tablet: real 0.010s user 0.002s sys 0.003s
I20240502 08:04:26.137434 11444 tablet_bootstrap.cc:492] T d117f9452626438cb18dda39df6eb6ed P f13c7769fc9f4bfa9796a5c29f715fb4: No bootstrap required, opened a new log
I20240502 08:04:26.137790 11445 tablet_bootstrap.cc:492] T f3666c62e67047bb9393d96859b539eb P c5e60e7e8697400c8c599205f6db37cf: Bootstrap starting.
I20240502 08:04:26.137959 11444 ts_tablet_manager.cc:1367] T d117f9452626438cb18dda39df6eb6ed P f13c7769fc9f4bfa9796a5c29f715fb4: Time spent bootstrapping tablet: real 0.014s user 0.013s sys 0.000s
I20240502 08:04:26.140807 11444 raft_consensus.cc:348] T d117f9452626438cb18dda39df6eb6ed P f13c7769fc9f4bfa9796a5c29f715fb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } }
I20240502 08:04:26.141604 11444 raft_consensus.cc:374] T d117f9452626438cb18dda39df6eb6ed P f13c7769fc9f4bfa9796a5c29f715fb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:26.141942 11444 raft_consensus.cc:729] T d117f9452626438cb18dda39df6eb6ed P f13c7769fc9f4bfa9796a5c29f715fb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f13c7769fc9f4bfa9796a5c29f715fb4, State: Initialized, Role: FOLLOWER
I20240502 08:04:26.142789 11444 consensus_queue.cc:260] T d117f9452626438cb18dda39df6eb6ed P f13c7769fc9f4bfa9796a5c29f715fb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } }
I20240502 08:04:26.143698 11445 tablet_bootstrap.cc:654] T f3666c62e67047bb9393d96859b539eb P c5e60e7e8697400c8c599205f6db37cf: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:26.145030 11444 ts_tablet_manager.cc:1398] T d117f9452626438cb18dda39df6eb6ed P f13c7769fc9f4bfa9796a5c29f715fb4: Time spent starting tablet: real 0.007s user 0.002s sys 0.002s
I20240502 08:04:26.145721 11443 ts_tablet_manager.cc:1398] T f3666c62e67047bb9393d96859b539eb P 122c848014b445728238d1ba49ac509c: Time spent starting tablet: real 0.014s user 0.005s sys 0.000s
I20240502 08:04:26.146037 11444 tablet_bootstrap.cc:492] T 8c2569709c52401cae6702e5e931ca83 P f13c7769fc9f4bfa9796a5c29f715fb4: Bootstrap starting.
I20240502 08:04:26.146744 11443 tablet_bootstrap.cc:492] T 743666c9f9a144978e534923cc63222e P 122c848014b445728238d1ba49ac509c: Bootstrap starting.
I20240502 08:04:26.153635 11444 tablet_bootstrap.cc:654] T 8c2569709c52401cae6702e5e931ca83 P f13c7769fc9f4bfa9796a5c29f715fb4: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:26.153640 11443 tablet_bootstrap.cc:654] T 743666c9f9a144978e534923cc63222e P 122c848014b445728238d1ba49ac509c: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:26.160802 11445 tablet_bootstrap.cc:492] T f3666c62e67047bb9393d96859b539eb P c5e60e7e8697400c8c599205f6db37cf: No bootstrap required, opened a new log
I20240502 08:04:26.161304 11445 ts_tablet_manager.cc:1367] T f3666c62e67047bb9393d96859b539eb P c5e60e7e8697400c8c599205f6db37cf: Time spent bootstrapping tablet: real 0.024s user 0.006s sys 0.010s
I20240502 08:04:26.161593 11443 tablet_bootstrap.cc:492] T 743666c9f9a144978e534923cc63222e P 122c848014b445728238d1ba49ac509c: No bootstrap required, opened a new log
I20240502 08:04:26.161780 11444 tablet_bootstrap.cc:492] T 8c2569709c52401cae6702e5e931ca83 P f13c7769fc9f4bfa9796a5c29f715fb4: No bootstrap required, opened a new log
I20240502 08:04:26.162070 11443 ts_tablet_manager.cc:1367] T 743666c9f9a144978e534923cc63222e P 122c848014b445728238d1ba49ac509c: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.001s
I20240502 08:04:26.162231 11444 ts_tablet_manager.cc:1367] T 8c2569709c52401cae6702e5e931ca83 P f13c7769fc9f4bfa9796a5c29f715fb4: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.007s
I20240502 08:04:26.163920 11445 raft_consensus.cc:348] T f3666c62e67047bb9393d96859b539eb P c5e60e7e8697400c8c599205f6db37cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } } peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } }
I20240502 08:04:26.164278 11444 raft_consensus.cc:348] T 8c2569709c52401cae6702e5e931ca83 P f13c7769fc9f4bfa9796a5c29f715fb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } } peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } }
I20240502 08:04:26.164700 11445 raft_consensus.cc:374] T f3666c62e67047bb9393d96859b539eb P c5e60e7e8697400c8c599205f6db37cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:26.164928 11444 raft_consensus.cc:374] T 8c2569709c52401cae6702e5e931ca83 P f13c7769fc9f4bfa9796a5c29f715fb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:26.165153 11445 raft_consensus.cc:729] T f3666c62e67047bb9393d96859b539eb P c5e60e7e8697400c8c599205f6db37cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e60e7e8697400c8c599205f6db37cf, State: Initialized, Role: FOLLOWER
I20240502 08:04:26.164749 11443 raft_consensus.cc:348] T 743666c9f9a144978e534923cc63222e P 122c848014b445728238d1ba49ac509c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } } peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } }
I20240502 08:04:26.165324 11444 raft_consensus.cc:729] T 8c2569709c52401cae6702e5e931ca83 P f13c7769fc9f4bfa9796a5c29f715fb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f13c7769fc9f4bfa9796a5c29f715fb4, State: Initialized, Role: FOLLOWER
I20240502 08:04:26.165633 11443 raft_consensus.cc:374] T 743666c9f9a144978e534923cc63222e P 122c848014b445728238d1ba49ac509c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:26.165983 11443 raft_consensus.cc:729] T 743666c9f9a144978e534923cc63222e P 122c848014b445728238d1ba49ac509c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 122c848014b445728238d1ba49ac509c, State: Initialized, Role: FOLLOWER
I20240502 08:04:26.166050 11444 consensus_queue.cc:260] T 8c2569709c52401cae6702e5e931ca83 P f13c7769fc9f4bfa9796a5c29f715fb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } } peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } }
I20240502 08:04:26.165938 11445 consensus_queue.cc:260] T f3666c62e67047bb9393d96859b539eb P c5e60e7e8697400c8c599205f6db37cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } } peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } }
I20240502 08:04:26.167021 11443 consensus_queue.cc:260] T 743666c9f9a144978e534923cc63222e P 122c848014b445728238d1ba49ac509c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } } peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } }
I20240502 08:04:26.168310 11444 ts_tablet_manager.cc:1398] T 8c2569709c52401cae6702e5e931ca83 P f13c7769fc9f4bfa9796a5c29f715fb4: Time spent starting tablet: real 0.006s user 0.001s sys 0.003s
I20240502 08:04:26.169318 11444 tablet_bootstrap.cc:492] T 743666c9f9a144978e534923cc63222e P f13c7769fc9f4bfa9796a5c29f715fb4: Bootstrap starting.
I20240502 08:04:26.171054 11445 ts_tablet_manager.cc:1398] T f3666c62e67047bb9393d96859b539eb P c5e60e7e8697400c8c599205f6db37cf: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20240502 08:04:26.171406 11443 ts_tablet_manager.cc:1398] T 743666c9f9a144978e534923cc63222e P 122c848014b445728238d1ba49ac509c: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20240502 08:04:26.172144 11445 tablet_bootstrap.cc:492] T 8c2569709c52401cae6702e5e931ca83 P c5e60e7e8697400c8c599205f6db37cf: Bootstrap starting.
I20240502 08:04:26.172343 11443 tablet_bootstrap.cc:492] T 8c2569709c52401cae6702e5e931ca83 P 122c848014b445728238d1ba49ac509c: Bootstrap starting.
I20240502 08:04:26.176600 11444 tablet_bootstrap.cc:654] T 743666c9f9a144978e534923cc63222e P f13c7769fc9f4bfa9796a5c29f715fb4: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:26.179997 11443 tablet_bootstrap.cc:654] T 8c2569709c52401cae6702e5e931ca83 P 122c848014b445728238d1ba49ac509c: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:26.187642 11445 tablet_bootstrap.cc:654] T 8c2569709c52401cae6702e5e931ca83 P c5e60e7e8697400c8c599205f6db37cf: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:26.189215 11444 tablet_bootstrap.cc:492] T 743666c9f9a144978e534923cc63222e P f13c7769fc9f4bfa9796a5c29f715fb4: No bootstrap required, opened a new log
I20240502 08:04:26.189791 11444 ts_tablet_manager.cc:1367] T 743666c9f9a144978e534923cc63222e P f13c7769fc9f4bfa9796a5c29f715fb4: Time spent bootstrapping tablet: real 0.021s user 0.015s sys 0.002s
I20240502 08:04:26.192761 11444 raft_consensus.cc:348] T 743666c9f9a144978e534923cc63222e P f13c7769fc9f4bfa9796a5c29f715fb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } } peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } }
I20240502 08:04:26.193663 11444 raft_consensus.cc:374] T 743666c9f9a144978e534923cc63222e P f13c7769fc9f4bfa9796a5c29f715fb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:26.193995 11444 raft_consensus.cc:729] T 743666c9f9a144978e534923cc63222e P f13c7769fc9f4bfa9796a5c29f715fb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f13c7769fc9f4bfa9796a5c29f715fb4, State: Initialized, Role: FOLLOWER
I20240502 08:04:26.195827 11444 consensus_queue.cc:260] T 743666c9f9a144978e534923cc63222e P f13c7769fc9f4bfa9796a5c29f715fb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } } peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } }
I20240502 08:04:26.196935 11443 tablet_bootstrap.cc:492] T 8c2569709c52401cae6702e5e931ca83 P 122c848014b445728238d1ba49ac509c: No bootstrap required, opened a new log
I20240502 08:04:26.197149 11445 tablet_bootstrap.cc:492] T 8c2569709c52401cae6702e5e931ca83 P c5e60e7e8697400c8c599205f6db37cf: No bootstrap required, opened a new log
I20240502 08:04:26.197513 11443 ts_tablet_manager.cc:1367] T 8c2569709c52401cae6702e5e931ca83 P 122c848014b445728238d1ba49ac509c: Time spent bootstrapping tablet: real 0.025s user 0.009s sys 0.014s
I20240502 08:04:26.197638 11445 ts_tablet_manager.cc:1367] T 8c2569709c52401cae6702e5e931ca83 P c5e60e7e8697400c8c599205f6db37cf: Time spent bootstrapping tablet: real 0.026s user 0.012s sys 0.003s
I20240502 08:04:26.198169 11444 ts_tablet_manager.cc:1398] T 743666c9f9a144978e534923cc63222e P f13c7769fc9f4bfa9796a5c29f715fb4: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20240502 08:04:26.200323 11443 raft_consensus.cc:348] T 8c2569709c52401cae6702e5e931ca83 P 122c848014b445728238d1ba49ac509c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } } peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } }
I20240502 08:04:26.200538 11445 raft_consensus.cc:348] T 8c2569709c52401cae6702e5e931ca83 P c5e60e7e8697400c8c599205f6db37cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } } peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } }
I20240502 08:04:26.201128 11443 raft_consensus.cc:374] T 8c2569709c52401cae6702e5e931ca83 P 122c848014b445728238d1ba49ac509c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:26.201290 11445 raft_consensus.cc:374] T 8c2569709c52401cae6702e5e931ca83 P c5e60e7e8697400c8c599205f6db37cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:26.201529 11443 raft_consensus.cc:729] T 8c2569709c52401cae6702e5e931ca83 P 122c848014b445728238d1ba49ac509c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 122c848014b445728238d1ba49ac509c, State: Initialized, Role: FOLLOWER
I20240502 08:04:26.201659 11445 raft_consensus.cc:729] T 8c2569709c52401cae6702e5e931ca83 P c5e60e7e8697400c8c599205f6db37cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e60e7e8697400c8c599205f6db37cf, State: Initialized, Role: FOLLOWER
I20240502 08:04:26.202330 11443 consensus_queue.cc:260] T 8c2569709c52401cae6702e5e931ca83 P 122c848014b445728238d1ba49ac509c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } } peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } }
I20240502 08:04:26.202441 11445 consensus_queue.cc:260] T 8c2569709c52401cae6702e5e931ca83 P c5e60e7e8697400c8c599205f6db37cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } } peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } }
I20240502 08:04:26.204733 11443 ts_tablet_manager.cc:1398] T 8c2569709c52401cae6702e5e931ca83 P 122c848014b445728238d1ba49ac509c: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240502 08:04:26.206522 11445 ts_tablet_manager.cc:1398] T 8c2569709c52401cae6702e5e931ca83 P c5e60e7e8697400c8c599205f6db37cf: Time spent starting tablet: real 0.009s user 0.005s sys 0.001s
I20240502 08:04:26.222612 11450 raft_consensus.cc:482] T f3666c62e67047bb9393d96859b539eb P c5e60e7e8697400c8c599205f6db37cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:04:26.223042 11450 raft_consensus.cc:504] T f3666c62e67047bb9393d96859b539eb P c5e60e7e8697400c8c599205f6db37cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } } peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } }
I20240502 08:04:26.224946 11461 raft_consensus.cc:482] T 8c2569709c52401cae6702e5e931ca83 P c5e60e7e8697400c8c599205f6db37cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:04:26.225543 11461 raft_consensus.cc:504] T 8c2569709c52401cae6702e5e931ca83 P c5e60e7e8697400c8c599205f6db37cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } } peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } }
I20240502 08:04:26.224992 11450 leader_election.cc:288] T f3666c62e67047bb9393d96859b539eb P c5e60e7e8697400c8c599205f6db37cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 122c848014b445728238d1ba49ac509c (127.6.204.194:44945), f13c7769fc9f4bfa9796a5c29f715fb4 (127.6.204.195:46495)
I20240502 08:04:26.228391 11461 leader_election.cc:288] T 8c2569709c52401cae6702e5e931ca83 P c5e60e7e8697400c8c599205f6db37cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 122c848014b445728238d1ba49ac509c (127.6.204.194:44945), f13c7769fc9f4bfa9796a5c29f715fb4 (127.6.204.195:46495)
I20240502 08:04:26.236949 11324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3666c62e67047bb9393d96859b539eb" candidate_uuid: "c5e60e7e8697400c8c599205f6db37cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "122c848014b445728238d1ba49ac509c" is_pre_election: true
I20240502 08:04:26.237305 11323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c2569709c52401cae6702e5e931ca83" candidate_uuid: "c5e60e7e8697400c8c599205f6db37cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "122c848014b445728238d1ba49ac509c" is_pre_election: true
I20240502 08:04:26.237390 11398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3666c62e67047bb9393d96859b539eb" candidate_uuid: "c5e60e7e8697400c8c599205f6db37cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" is_pre_election: true
I20240502 08:04:26.237789 11397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c2569709c52401cae6702e5e931ca83" candidate_uuid: "c5e60e7e8697400c8c599205f6db37cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" is_pre_election: true
I20240502 08:04:26.237903 11324 raft_consensus.cc:2454] T f3666c62e67047bb9393d96859b539eb P 122c848014b445728238d1ba49ac509c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5e60e7e8697400c8c599205f6db37cf in term 0.
I20240502 08:04:26.238224 11323 raft_consensus.cc:2454] T 8c2569709c52401cae6702e5e931ca83 P 122c848014b445728238d1ba49ac509c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5e60e7e8697400c8c599205f6db37cf in term 0.
I20240502 08:04:26.238433 11398 raft_consensus.cc:2454] T f3666c62e67047bb9393d96859b539eb P f13c7769fc9f4bfa9796a5c29f715fb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5e60e7e8697400c8c599205f6db37cf in term 0.
I20240502 08:04:26.238523 11397 raft_consensus.cc:2454] T 8c2569709c52401cae6702e5e931ca83 P f13c7769fc9f4bfa9796a5c29f715fb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5e60e7e8697400c8c599205f6db37cf in term 0.
I20240502 08:04:26.239466 11215 leader_election.cc:302] T f3666c62e67047bb9393d96859b539eb P c5e60e7e8697400c8c599205f6db37cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 122c848014b445728238d1ba49ac509c, c5e60e7e8697400c8c599205f6db37cf; no voters:
I20240502 08:04:26.240234 11461 raft_consensus.cc:2781] T f3666c62e67047bb9393d96859b539eb P c5e60e7e8697400c8c599205f6db37cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:04:26.240370 11213 leader_election.cc:302] T 8c2569709c52401cae6702e5e931ca83 P c5e60e7e8697400c8c599205f6db37cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5e60e7e8697400c8c599205f6db37cf, f13c7769fc9f4bfa9796a5c29f715fb4; no voters:
I20240502 08:04:26.240634 11461 raft_consensus.cc:482] T f3666c62e67047bb9393d96859b539eb P c5e60e7e8697400c8c599205f6db37cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:04:26.240986 11461 raft_consensus.cc:3037] T f3666c62e67047bb9393d96859b539eb P c5e60e7e8697400c8c599205f6db37cf [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:26.241161 11450 raft_consensus.cc:2781] T 8c2569709c52401cae6702e5e931ca83 P c5e60e7e8697400c8c599205f6db37cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:04:26.241535 11450 raft_consensus.cc:482] T 8c2569709c52401cae6702e5e931ca83 P c5e60e7e8697400c8c599205f6db37cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:04:26.241881 11450 raft_consensus.cc:3037] T 8c2569709c52401cae6702e5e931ca83 P c5e60e7e8697400c8c599205f6db37cf [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:26.248504 11450 raft_consensus.cc:504] T 8c2569709c52401cae6702e5e931ca83 P c5e60e7e8697400c8c599205f6db37cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } } peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } }
I20240502 08:04:26.248417 11461 raft_consensus.cc:504] T f3666c62e67047bb9393d96859b539eb P c5e60e7e8697400c8c599205f6db37cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } } peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } }
I20240502 08:04:26.250514 11450 leader_election.cc:288] T 8c2569709c52401cae6702e5e931ca83 P c5e60e7e8697400c8c599205f6db37cf [CANDIDATE]: Term 1 election: Requested vote from peers 122c848014b445728238d1ba49ac509c (127.6.204.194:44945), f13c7769fc9f4bfa9796a5c29f715fb4 (127.6.204.195:46495)
I20240502 08:04:26.251291 11323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3666c62e67047bb9393d96859b539eb" candidate_uuid: "c5e60e7e8697400c8c599205f6db37cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "122c848014b445728238d1ba49ac509c"
I20240502 08:04:26.251596 11397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3666c62e67047bb9393d96859b539eb" candidate_uuid: "c5e60e7e8697400c8c599205f6db37cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4"
I20240502 08:04:26.251638 11324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c2569709c52401cae6702e5e931ca83" candidate_uuid: "c5e60e7e8697400c8c599205f6db37cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "122c848014b445728238d1ba49ac509c"
I20240502 08:04:26.251935 11398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c2569709c52401cae6702e5e931ca83" candidate_uuid: "c5e60e7e8697400c8c599205f6db37cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4"
I20240502 08:04:26.252060 11323 raft_consensus.cc:3037] T f3666c62e67047bb9393d96859b539eb P 122c848014b445728238d1ba49ac509c [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:26.252189 11397 raft_consensus.cc:3037] T f3666c62e67047bb9393d96859b539eb P f13c7769fc9f4bfa9796a5c29f715fb4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:26.252403 11324 raft_consensus.cc:3037] T 8c2569709c52401cae6702e5e931ca83 P 122c848014b445728238d1ba49ac509c [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:26.252552 11398 raft_consensus.cc:3037] T 8c2569709c52401cae6702e5e931ca83 P f13c7769fc9f4bfa9796a5c29f715fb4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:26.259286 11397 raft_consensus.cc:2454] T f3666c62e67047bb9393d96859b539eb P f13c7769fc9f4bfa9796a5c29f715fb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5e60e7e8697400c8c599205f6db37cf in term 1.
I20240502 08:04:26.259470 11324 raft_consensus.cc:2454] T 8c2569709c52401cae6702e5e931ca83 P 122c848014b445728238d1ba49ac509c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5e60e7e8697400c8c599205f6db37cf in term 1.
I20240502 08:04:26.250507 11461 leader_election.cc:288] T f3666c62e67047bb9393d96859b539eb P c5e60e7e8697400c8c599205f6db37cf [CANDIDATE]: Term 1 election: Requested vote from peers 122c848014b445728238d1ba49ac509c (127.6.204.194:44945), f13c7769fc9f4bfa9796a5c29f715fb4 (127.6.204.195:46495)
I20240502 08:04:26.259948 11449 raft_consensus.cc:482] T 743666c9f9a144978e534923cc63222e P 122c848014b445728238d1ba49ac509c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:04:26.259291 11323 raft_consensus.cc:2454] T f3666c62e67047bb9393d96859b539eb P 122c848014b445728238d1ba49ac509c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5e60e7e8697400c8c599205f6db37cf in term 1.
I20240502 08:04:26.260533 11449 raft_consensus.cc:504] T 743666c9f9a144978e534923cc63222e P 122c848014b445728238d1ba49ac509c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } } peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } }
I20240502 08:04:26.260905 11213 leader_election.cc:302] T f3666c62e67047bb9393d96859b539eb P c5e60e7e8697400c8c599205f6db37cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5e60e7e8697400c8c599205f6db37cf, f13c7769fc9f4bfa9796a5c29f715fb4; no voters:
I20240502 08:04:26.261080 11215 leader_election.cc:302] T 8c2569709c52401cae6702e5e931ca83 P c5e60e7e8697400c8c599205f6db37cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 122c848014b445728238d1ba49ac509c, c5e60e7e8697400c8c599205f6db37cf; no voters:
I20240502 08:04:26.259470 11398 raft_consensus.cc:2454] T 8c2569709c52401cae6702e5e931ca83 P f13c7769fc9f4bfa9796a5c29f715fb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5e60e7e8697400c8c599205f6db37cf in term 1.
I20240502 08:04:26.261775 11461 raft_consensus.cc:2781] T f3666c62e67047bb9393d96859b539eb P c5e60e7e8697400c8c599205f6db37cf [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:04:26.262549 11450 raft_consensus.cc:2781] T 8c2569709c52401cae6702e5e931ca83 P c5e60e7e8697400c8c599205f6db37cf [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:04:26.264644 11449 leader_election.cc:288] T 743666c9f9a144978e534923cc63222e P 122c848014b445728238d1ba49ac509c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5e60e7e8697400c8c599205f6db37cf (127.6.204.193:39769), f13c7769fc9f4bfa9796a5c29f715fb4 (127.6.204.195:46495)
I20240502 08:04:26.267324 11450 raft_consensus.cc:686] T 8c2569709c52401cae6702e5e931ca83 P c5e60e7e8697400c8c599205f6db37cf [term 1 LEADER]: Becoming Leader. State: Replica: c5e60e7e8697400c8c599205f6db37cf, State: Running, Role: LEADER
I20240502 08:04:26.268092 11450 consensus_queue.cc:237] T 8c2569709c52401cae6702e5e931ca83 P c5e60e7e8697400c8c599205f6db37cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } } peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } }
I20240502 08:04:26.271306 11461 raft_consensus.cc:686] T f3666c62e67047bb9393d96859b539eb P c5e60e7e8697400c8c599205f6db37cf [term 1 LEADER]: Becoming Leader. State: Replica: c5e60e7e8697400c8c599205f6db37cf, State: Running, Role: LEADER
I20240502 08:04:26.272266 11461 consensus_queue.cc:237] T f3666c62e67047bb9393d96859b539eb P c5e60e7e8697400c8c599205f6db37cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } } peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } }
I20240502 08:04:26.283921 11249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "743666c9f9a144978e534923cc63222e" candidate_uuid: "122c848014b445728238d1ba49ac509c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e60e7e8697400c8c599205f6db37cf" is_pre_election: true
I20240502 08:04:26.284713 11249 raft_consensus.cc:2454] T 743666c9f9a144978e534923cc63222e P c5e60e7e8697400c8c599205f6db37cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 122c848014b445728238d1ba49ac509c in term 0.
I20240502 08:04:26.285949 11288 leader_election.cc:302] T 743666c9f9a144978e534923cc63222e P 122c848014b445728238d1ba49ac509c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 122c848014b445728238d1ba49ac509c, c5e60e7e8697400c8c599205f6db37cf; no voters:
I20240502 08:04:26.285218 11146 catalog_manager.cc:5484] T f3666c62e67047bb9393d96859b539eb P c5e60e7e8697400c8c599205f6db37cf reported cstate change: term changed from 0 to 1, leader changed from <none> to c5e60e7e8697400c8c599205f6db37cf (127.6.204.193). New cstate: current_term: 1 leader_uuid: "c5e60e7e8697400c8c599205f6db37cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } health_report { overall_health: UNKNOWN } } }
I20240502 08:04:26.286913 11449 raft_consensus.cc:2781] T 743666c9f9a144978e534923cc63222e P 122c848014b445728238d1ba49ac509c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:04:26.287281 11449 raft_consensus.cc:482] T 743666c9f9a144978e534923cc63222e P 122c848014b445728238d1ba49ac509c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:04:26.287637 11449 raft_consensus.cc:3037] T 743666c9f9a144978e534923cc63222e P 122c848014b445728238d1ba49ac509c [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:26.287464 11146 catalog_manager.cc:5484] T 8c2569709c52401cae6702e5e931ca83 P c5e60e7e8697400c8c599205f6db37cf reported cstate change: term changed from 0 to 1, leader changed from <none> to c5e60e7e8697400c8c599205f6db37cf (127.6.204.193). New cstate: current_term: 1 leader_uuid: "c5e60e7e8697400c8c599205f6db37cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } health_report { overall_health: UNKNOWN } } }
I20240502 08:04:26.289096 11398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "743666c9f9a144978e534923cc63222e" candidate_uuid: "122c848014b445728238d1ba49ac509c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" is_pre_election: true
I20240502 08:04:26.289616 11398 raft_consensus.cc:2454] T 743666c9f9a144978e534923cc63222e P f13c7769fc9f4bfa9796a5c29f715fb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 122c848014b445728238d1ba49ac509c in term 0.
I20240502 08:04:26.293378 11449 raft_consensus.cc:504] T 743666c9f9a144978e534923cc63222e P 122c848014b445728238d1ba49ac509c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } } peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } }
I20240502 08:04:26.295188 11449 leader_election.cc:288] T 743666c9f9a144978e534923cc63222e P 122c848014b445728238d1ba49ac509c [CANDIDATE]: Term 1 election: Requested vote from peers c5e60e7e8697400c8c599205f6db37cf (127.6.204.193:39769), f13c7769fc9f4bfa9796a5c29f715fb4 (127.6.204.195:46495)
I20240502 08:04:26.296085 11398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "743666c9f9a144978e534923cc63222e" candidate_uuid: "122c848014b445728238d1ba49ac509c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4"
I20240502 08:04:26.296110 11249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "743666c9f9a144978e534923cc63222e" candidate_uuid: "122c848014b445728238d1ba49ac509c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e60e7e8697400c8c599205f6db37cf"
I20240502 08:04:26.296676 11398 raft_consensus.cc:3037] T 743666c9f9a144978e534923cc63222e P f13c7769fc9f4bfa9796a5c29f715fb4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:26.296860 11249 raft_consensus.cc:3037] T 743666c9f9a144978e534923cc63222e P c5e60e7e8697400c8c599205f6db37cf [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:26.303215 11249 raft_consensus.cc:2454] T 743666c9f9a144978e534923cc63222e P c5e60e7e8697400c8c599205f6db37cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 122c848014b445728238d1ba49ac509c in term 1.
I20240502 08:04:26.303234 11398 raft_consensus.cc:2454] T 743666c9f9a144978e534923cc63222e P f13c7769fc9f4bfa9796a5c29f715fb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 122c848014b445728238d1ba49ac509c in term 1.
I20240502 08:04:26.304322 11288 leader_election.cc:302] T 743666c9f9a144978e534923cc63222e P 122c848014b445728238d1ba49ac509c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 122c848014b445728238d1ba49ac509c, c5e60e7e8697400c8c599205f6db37cf; no voters:
I20240502 08:04:26.305100 11449 raft_consensus.cc:2781] T 743666c9f9a144978e534923cc63222e P 122c848014b445728238d1ba49ac509c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:04:26.306281 11449 raft_consensus.cc:686] T 743666c9f9a144978e534923cc63222e P 122c848014b445728238d1ba49ac509c [term 1 LEADER]: Becoming Leader. State: Replica: 122c848014b445728238d1ba49ac509c, State: Running, Role: LEADER
I20240502 08:04:26.307092 11449 consensus_queue.cc:237] T 743666c9f9a144978e534923cc63222e P 122c848014b445728238d1ba49ac509c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } } peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } }
I20240502 08:04:26.313812 11146 catalog_manager.cc:5484] T 743666c9f9a144978e534923cc63222e P 122c848014b445728238d1ba49ac509c reported cstate change: term changed from 0 to 1, leader changed from <none> to 122c848014b445728238d1ba49ac509c (127.6.204.194). New cstate: current_term: 1 leader_uuid: "122c848014b445728238d1ba49ac509c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } health_report { overall_health: UNKNOWN } } }
I20240502 08:04:26.409096 11449 raft_consensus.cc:482] T d117f9452626438cb18dda39df6eb6ed P 122c848014b445728238d1ba49ac509c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:04:26.409579 11449 raft_consensus.cc:504] T d117f9452626438cb18dda39df6eb6ed P 122c848014b445728238d1ba49ac509c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } }
I20240502 08:04:26.411327 11449 leader_election.cc:288] T d117f9452626438cb18dda39df6eb6ed P 122c848014b445728238d1ba49ac509c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5e60e7e8697400c8c599205f6db37cf (127.6.204.193:39769), f13c7769fc9f4bfa9796a5c29f715fb4 (127.6.204.195:46495)
I20240502 08:04:26.412149 11249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d117f9452626438cb18dda39df6eb6ed" candidate_uuid: "122c848014b445728238d1ba49ac509c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e60e7e8697400c8c599205f6db37cf" is_pre_election: true
I20240502 08:04:26.412307 11398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d117f9452626438cb18dda39df6eb6ed" candidate_uuid: "122c848014b445728238d1ba49ac509c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" is_pre_election: true
I20240502 08:04:26.412753 11249 raft_consensus.cc:2454] T d117f9452626438cb18dda39df6eb6ed P c5e60e7e8697400c8c599205f6db37cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 122c848014b445728238d1ba49ac509c in term 0.
I20240502 08:04:26.412859 11398 raft_consensus.cc:2454] T d117f9452626438cb18dda39df6eb6ed P f13c7769fc9f4bfa9796a5c29f715fb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 122c848014b445728238d1ba49ac509c in term 0.
I20240502 08:04:26.413600 11288 leader_election.cc:302] T d117f9452626438cb18dda39df6eb6ed P 122c848014b445728238d1ba49ac509c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 122c848014b445728238d1ba49ac509c, c5e60e7e8697400c8c599205f6db37cf; no voters:
I20240502 08:04:26.414305 11449 raft_consensus.cc:2781] T d117f9452626438cb18dda39df6eb6ed P 122c848014b445728238d1ba49ac509c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:04:26.414580 11449 raft_consensus.cc:482] T d117f9452626438cb18dda39df6eb6ed P 122c848014b445728238d1ba49ac509c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:04:26.414822 11449 raft_consensus.cc:3037] T d117f9452626438cb18dda39df6eb6ed P 122c848014b445728238d1ba49ac509c [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:26.419705 11449 raft_consensus.cc:504] T d117f9452626438cb18dda39df6eb6ed P 122c848014b445728238d1ba49ac509c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } }
I20240502 08:04:26.421115 11449 leader_election.cc:288] T d117f9452626438cb18dda39df6eb6ed P 122c848014b445728238d1ba49ac509c [CANDIDATE]: Term 1 election: Requested vote from peers c5e60e7e8697400c8c599205f6db37cf (127.6.204.193:39769), f13c7769fc9f4bfa9796a5c29f715fb4 (127.6.204.195:46495)
I20240502 08:04:26.421976 11249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d117f9452626438cb18dda39df6eb6ed" candidate_uuid: "122c848014b445728238d1ba49ac509c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e60e7e8697400c8c599205f6db37cf"
I20240502 08:04:26.422104 11398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d117f9452626438cb18dda39df6eb6ed" candidate_uuid: "122c848014b445728238d1ba49ac509c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4"
I20240502 08:04:26.422535 11249 raft_consensus.cc:3037] T d117f9452626438cb18dda39df6eb6ed P c5e60e7e8697400c8c599205f6db37cf [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:26.422618 11398 raft_consensus.cc:3037] T d117f9452626438cb18dda39df6eb6ed P f13c7769fc9f4bfa9796a5c29f715fb4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:26.427160 11398 raft_consensus.cc:2454] T d117f9452626438cb18dda39df6eb6ed P f13c7769fc9f4bfa9796a5c29f715fb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 122c848014b445728238d1ba49ac509c in term 1.
I20240502 08:04:26.427834 11249 raft_consensus.cc:2454] T d117f9452626438cb18dda39df6eb6ed P c5e60e7e8697400c8c599205f6db37cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 122c848014b445728238d1ba49ac509c in term 1.
I20240502 08:04:26.428063 11288 leader_election.cc:302] T d117f9452626438cb18dda39df6eb6ed P 122c848014b445728238d1ba49ac509c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 122c848014b445728238d1ba49ac509c, f13c7769fc9f4bfa9796a5c29f715fb4; no voters:
I20240502 08:04:26.428795 11449 raft_consensus.cc:2781] T d117f9452626438cb18dda39df6eb6ed P 122c848014b445728238d1ba49ac509c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:04:26.429147 11449 raft_consensus.cc:686] T d117f9452626438cb18dda39df6eb6ed P 122c848014b445728238d1ba49ac509c [term 1 LEADER]: Becoming Leader. State: Replica: 122c848014b445728238d1ba49ac509c, State: Running, Role: LEADER
I20240502 08:04:26.429719 11449 consensus_queue.cc:237] T d117f9452626438cb18dda39df6eb6ed P 122c848014b445728238d1ba49ac509c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } }
I20240502 08:04:26.436417 11146 catalog_manager.cc:5484] T d117f9452626438cb18dda39df6eb6ed P 122c848014b445728238d1ba49ac509c reported cstate change: term changed from 0 to 1, leader changed from <none> to 122c848014b445728238d1ba49ac509c (127.6.204.194). New cstate: current_term: 1 leader_uuid: "122c848014b445728238d1ba49ac509c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } health_report { overall_health: UNKNOWN } } }
I20240502 08:04:26.463755 6963 tablet_server.cc:178] TabletServer@127.6.204.193:0 shutting down...
I20240502 08:04:26.479339 6963 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 08:04:26.480129 6963 tablet_replica.cc:330] T 8c2569709c52401cae6702e5e931ca83 P c5e60e7e8697400c8c599205f6db37cf: stopping tablet replica
I20240502 08:04:26.480707 6963 raft_consensus.cc:2229] T 8c2569709c52401cae6702e5e931ca83 P c5e60e7e8697400c8c599205f6db37cf [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:04:26.481292 6963 pending_rounds.cc:62] T 8c2569709c52401cae6702e5e931ca83 P c5e60e7e8697400c8c599205f6db37cf: Trying to abort 1 pending ops.
I20240502 08:04:26.481490 6963 pending_rounds.cc:68] T 8c2569709c52401cae6702e5e931ca83 P c5e60e7e8697400c8c599205f6db37cf: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023153423441457152 op_type: NO_OP noop_request { }
I20240502 08:04:26.481803 6963 raft_consensus.cc:2866] T 8c2569709c52401cae6702e5e931ca83 P c5e60e7e8697400c8c599205f6db37cf [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240502 08:04:26.482067 6963 raft_consensus.cc:2258] T 8c2569709c52401cae6702e5e931ca83 P c5e60e7e8697400c8c599205f6db37cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:04:26.484330 6963 tablet_replica.cc:330] T f3666c62e67047bb9393d96859b539eb P c5e60e7e8697400c8c599205f6db37cf: stopping tablet replica
I20240502 08:04:26.484822 6963 raft_consensus.cc:2229] T f3666c62e67047bb9393d96859b539eb P c5e60e7e8697400c8c599205f6db37cf [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:04:26.485335 6963 pending_rounds.cc:62] T f3666c62e67047bb9393d96859b539eb P c5e60e7e8697400c8c599205f6db37cf: Trying to abort 1 pending ops.
I20240502 08:04:26.485491 6963 pending_rounds.cc:68] T f3666c62e67047bb9393d96859b539eb P c5e60e7e8697400c8c599205f6db37cf: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023153423460667392 op_type: NO_OP noop_request { }
I20240502 08:04:26.485750 6963 raft_consensus.cc:2866] T f3666c62e67047bb9393d96859b539eb P c5e60e7e8697400c8c599205f6db37cf [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240502 08:04:26.485985 6963 raft_consensus.cc:2258] T f3666c62e67047bb9393d96859b539eb P c5e60e7e8697400c8c599205f6db37cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:04:26.488022 6963 tablet_replica.cc:330] T 743666c9f9a144978e534923cc63222e P c5e60e7e8697400c8c599205f6db37cf: stopping tablet replica
I20240502 08:04:26.488487 6963 raft_consensus.cc:2229] T 743666c9f9a144978e534923cc63222e P c5e60e7e8697400c8c599205f6db37cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:04:26.488824 6963 raft_consensus.cc:2258] T 743666c9f9a144978e534923cc63222e P c5e60e7e8697400c8c599205f6db37cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:04:26.490506 6963 tablet_replica.cc:330] T d117f9452626438cb18dda39df6eb6ed P c5e60e7e8697400c8c599205f6db37cf: stopping tablet replica
I20240502 08:04:26.490926 6963 raft_consensus.cc:2229] T d117f9452626438cb18dda39df6eb6ed P c5e60e7e8697400c8c599205f6db37cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:04:26.491247 6963 raft_consensus.cc:2258] T d117f9452626438cb18dda39df6eb6ed P c5e60e7e8697400c8c599205f6db37cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:04:26.513587 6963 tablet_server.cc:194] TabletServer@127.6.204.193:0 shutdown complete.
I20240502 08:04:26.792960 11449 consensus_queue.cc:1035] T 743666c9f9a144978e534923cc63222e P 122c848014b445728238d1ba49ac509c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:04:26.805182 11288 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.204.193:39769: connect: Connection refused (error 111) [suppressed 6 similar messages]
W20240502 08:04:26.809602 11288 consensus_peers.cc:485] T 743666c9f9a144978e534923cc63222e P 122c848014b445728238d1ba49ac509c -> Peer c5e60e7e8697400c8c599205f6db37cf (127.6.204.193:39769): Couldn't send request to peer c5e60e7e8697400c8c599205f6db37cf. Status: Network error: Client connection negotiation failed: client connection to 127.6.204.193:39769: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:04:26.903620 11449 consensus_queue.cc:1035] T d117f9452626438cb18dda39df6eb6ed P 122c848014b445728238d1ba49ac509c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:04:26.915988 11288 consensus_peers.cc:485] T d117f9452626438cb18dda39df6eb6ed P 122c848014b445728238d1ba49ac509c -> Peer c5e60e7e8697400c8c599205f6db37cf (127.6.204.193:39769): Couldn't send request to peer c5e60e7e8697400c8c599205f6db37cf. Status: Network error: Client connection negotiation failed: client connection to 127.6.204.193:39769: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:04:27.523031 11195 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver c5e60e7e8697400c8c599205f6db37cf not available for placement
I20240502 08:04:27.527053 6963 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:04:27.533020 11481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:04:27.533872 11482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:04:27.535916 11484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:04:27.536247 6963 server_base.cc:1034] running on GCE node
I20240502 08:04:27.538069 6963 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:04:27.538322 6963 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:04:27.538491 6963 hybrid_clock.cc:648] HybridClock initialized: now 1714637067538466 us; error 0 us; skew 500 ppm
I20240502 08:04:27.538995 6963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:04:27.541219 6963 webserver.cc:460] Webserver started at http://127.6.204.196:37863/ using document root <none> and password file <none>
I20240502 08:04:27.541692 6963 fs_manager.cc:359] Metadata directory not provided
I20240502 08:04:27.541877 6963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:04:27.542129 6963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:04:27.543239 6963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714636933625983-6963-0/minicluster-data/ts-3-root/instance:
uuid: "f5b0428595be46d891987e38f4a83a15"
format_stamp: "Formatted at 2024-05-02 08:04:27 on dist-test-slave-32xs"
I20240502 08:04:27.547549 6963 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 08:04:27.550733 11489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:04:27.551529 6963 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 08:04:27.551810 6963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714636933625983-6963-0/minicluster-data/ts-3-root
uuid: "f5b0428595be46d891987e38f4a83a15"
format_stamp: "Formatted at 2024-05-02 08:04:27 on dist-test-slave-32xs"
I20240502 08:04:27.552093 6963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714636933625983-6963-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714636933625983-6963-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714636933625983-6963-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:04:27.560667 6963 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:04:27.561589 6963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:04:27.562947 6963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:04:27.565083 6963 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:04:27.565284 6963 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:04:27.565521 6963 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:04:27.565677 6963 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:04:27.602775 6963 rpc_server.cc:307] RPC server started. Bound to: 127.6.204.196:36925
I20240502 08:04:27.602881 11551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.204.196:36925 every 8 connection(s)
I20240502 08:04:27.616588 11552 heartbeater.cc:347] Connected to a master server at 127.6.204.254:37875
I20240502 08:04:27.616976 11552 heartbeater.cc:464] Registering TS with master...
I20240502 08:04:27.617738 11552 heartbeater.cc:511] Master 127.6.204.254:37875 requested a full tablet report, sending...
I20240502 08:04:27.619685 11145 ts_manager.cc:194] Registered new tserver with Master: f5b0428595be46d891987e38f4a83a15 (127.6.204.196:36925)
I20240502 08:04:27.621062 11145 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52826
I20240502 08:04:27.677889 11449 consensus_queue.cc:579] T 743666c9f9a144978e534923cc63222e P 122c848014b445728238d1ba49ac509c [LEADER]: Leader has been unable to successfully communicate with peer c5e60e7e8697400c8c599205f6db37cf for more than 1 seconds (1.370s)
I20240502 08:04:27.688177 11323 consensus_queue.cc:237] T 743666c9f9a144978e534923cc63222e P 122c848014b445728238d1ba49ac509c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } } peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } } peers { permanent_uuid: "f5b0428595be46d891987e38f4a83a15" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 36925 } attrs { promote: true } }
I20240502 08:04:27.694344 11398 raft_consensus.cc:1261] T 743666c9f9a144978e534923cc63222e P f13c7769fc9f4bfa9796a5c29f715fb4 [term 1 FOLLOWER]: Refusing update from remote peer 122c848014b445728238d1ba49ac509c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:04:27.696141 11556 consensus_queue.cc:1035] T 743666c9f9a144978e534923cc63222e P 122c848014b445728238d1ba49ac509c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 08:04:27.698505 11288 consensus_peers.cc:485] T 743666c9f9a144978e534923cc63222e P 122c848014b445728238d1ba49ac509c -> Peer c5e60e7e8697400c8c599205f6db37cf (127.6.204.193:39769): Couldn't send request to peer c5e60e7e8697400c8c599205f6db37cf. Status: Network error: Client connection negotiation failed: client connection to 127.6.204.193:39769: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:04:27.702169 11557 raft_consensus.cc:2932] T 743666c9f9a144978e534923cc63222e P 122c848014b445728238d1ba49ac509c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f5b0428595be46d891987e38f4a83a15 (127.6.204.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } } peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } } peers { permanent_uuid: "f5b0428595be46d891987e38f4a83a15" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 36925 } attrs { promote: true } } }
W20240502 08:04:27.704311 11288 consensus_peers.cc:485] T 743666c9f9a144978e534923cc63222e P 122c848014b445728238d1ba49ac509c -> Peer f5b0428595be46d891987e38f4a83a15 (127.6.204.196:36925): Couldn't send request to peer f5b0428595be46d891987e38f4a83a15. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 743666c9f9a144978e534923cc63222e. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:04:27.703920 11398 raft_consensus.cc:2932] T 743666c9f9a144978e534923cc63222e P f13c7769fc9f4bfa9796a5c29f715fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f5b0428595be46d891987e38f4a83a15 (127.6.204.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } } peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } } peers { permanent_uuid: "f5b0428595be46d891987e38f4a83a15" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 36925 } attrs { promote: true } } }
I20240502 08:04:27.711891 11134 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 743666c9f9a144978e534923cc63222e with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 08:04:27.716264 11145 catalog_manager.cc:5484] T 743666c9f9a144978e534923cc63222e P 122c848014b445728238d1ba49ac509c reported cstate change: config changed from index -1 to 2, NON_VOTER f5b0428595be46d891987e38f4a83a15 (127.6.204.196) added. New cstate: current_term: 1 leader_uuid: "122c848014b445728238d1ba49ac509c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5b0428595be46d891987e38f4a83a15" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 36925 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:04:27.765805 11560 consensus_queue.cc:579] T d117f9452626438cb18dda39df6eb6ed P 122c848014b445728238d1ba49ac509c [LEADER]: Leader has been unable to successfully communicate with peer c5e60e7e8697400c8c599205f6db37cf for more than 1 seconds (1.335s)
I20240502 08:04:27.774670 11323 consensus_queue.cc:237] T d117f9452626438cb18dda39df6eb6ed P 122c848014b445728238d1ba49ac509c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } } peers { permanent_uuid: "f5b0428595be46d891987e38f4a83a15" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 36925 } attrs { promote: true } }
I20240502 08:04:27.779608 11398 raft_consensus.cc:1261] T d117f9452626438cb18dda39df6eb6ed P f13c7769fc9f4bfa9796a5c29f715fb4 [term 1 FOLLOWER]: Refusing update from remote peer 122c848014b445728238d1ba49ac509c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 08:04:27.779701 11288 consensus_peers.cc:485] T d117f9452626438cb18dda39df6eb6ed P 122c848014b445728238d1ba49ac509c -> Peer f5b0428595be46d891987e38f4a83a15 (127.6.204.196:36925): Couldn't send request to peer f5b0428595be46d891987e38f4a83a15. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d117f9452626438cb18dda39df6eb6ed. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:04:27.780931 11560 consensus_queue.cc:1035] T d117f9452626438cb18dda39df6eb6ed P 122c848014b445728238d1ba49ac509c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 08:04:27.782034 11288 consensus_peers.cc:485] T d117f9452626438cb18dda39df6eb6ed P 122c848014b445728238d1ba49ac509c -> Peer c5e60e7e8697400c8c599205f6db37cf (127.6.204.193:39769): Couldn't send request to peer c5e60e7e8697400c8c599205f6db37cf. Status: Network error: Client connection negotiation failed: client connection to 127.6.204.193:39769: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:04:27.786391 11556 raft_consensus.cc:2932] T d117f9452626438cb18dda39df6eb6ed P 122c848014b445728238d1ba49ac509c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f5b0428595be46d891987e38f4a83a15 (127.6.204.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } } peers { permanent_uuid: "f5b0428595be46d891987e38f4a83a15" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 36925 } attrs { promote: true } } }
I20240502 08:04:27.787822 11398 raft_consensus.cc:2932] T d117f9452626438cb18dda39df6eb6ed P f13c7769fc9f4bfa9796a5c29f715fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f5b0428595be46d891987e38f4a83a15 (127.6.204.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } } peers { permanent_uuid: "f5b0428595be46d891987e38f4a83a15" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 36925 } attrs { promote: true } } }
I20240502 08:04:27.793011 11134 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet d117f9452626438cb18dda39df6eb6ed with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 08:04:27.796854 11146 catalog_manager.cc:5484] T d117f9452626438cb18dda39df6eb6ed P 122c848014b445728238d1ba49ac509c reported cstate change: config changed from index -1 to 2, NON_VOTER f5b0428595be46d891987e38f4a83a15 (127.6.204.196) added. New cstate: current_term: 1 leader_uuid: "122c848014b445728238d1ba49ac509c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5b0428595be46d891987e38f4a83a15" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 36925 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:04:27.958179 11561 raft_consensus.cc:482] T f3666c62e67047bb9393d96859b539eb P f13c7769fc9f4bfa9796a5c29f715fb4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:04:27.958694 11561 raft_consensus.cc:504] T f3666c62e67047bb9393d96859b539eb P f13c7769fc9f4bfa9796a5c29f715fb4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } } peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } }
I20240502 08:04:27.960577 11561 leader_election.cc:288] T f3666c62e67047bb9393d96859b539eb P f13c7769fc9f4bfa9796a5c29f715fb4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5e60e7e8697400c8c599205f6db37cf (127.6.204.193:39769), 122c848014b445728238d1ba49ac509c (127.6.204.194:44945)
I20240502 08:04:27.966264 11560 raft_consensus.cc:482] T f3666c62e67047bb9393d96859b539eb P 122c848014b445728238d1ba49ac509c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240502 08:04:27.966840 11362 leader_election.cc:334] T f3666c62e67047bb9393d96859b539eb P f13c7769fc9f4bfa9796a5c29f715fb4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c5e60e7e8697400c8c599205f6db37cf (127.6.204.193:39769): Network error: Client connection negotiation failed: client connection to 127.6.204.193:39769: connect: Connection refused (error 111)
I20240502 08:04:27.966794 11560 raft_consensus.cc:504] T f3666c62e67047bb9393d96859b539eb P 122c848014b445728238d1ba49ac509c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } } peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } }
I20240502 08:04:27.968643 11560 leader_election.cc:288] T f3666c62e67047bb9393d96859b539eb P 122c848014b445728238d1ba49ac509c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5e60e7e8697400c8c599205f6db37cf (127.6.204.193:39769), f13c7769fc9f4bfa9796a5c29f715fb4 (127.6.204.195:46495)
I20240502 08:04:27.970199 11398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3666c62e67047bb9393d96859b539eb" candidate_uuid: "122c848014b445728238d1ba49ac509c" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" is_pre_election: true
I20240502 08:04:27.970880 11398 raft_consensus.cc:2454] T f3666c62e67047bb9393d96859b539eb P f13c7769fc9f4bfa9796a5c29f715fb4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 122c848014b445728238d1ba49ac509c in term 1.
I20240502 08:04:27.971971 11288 leader_election.cc:302] T f3666c62e67047bb9393d96859b539eb P 122c848014b445728238d1ba49ac509c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 122c848014b445728238d1ba49ac509c, f13c7769fc9f4bfa9796a5c29f715fb4; no voters:
I20240502 08:04:27.972405 11323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3666c62e67047bb9393d96859b539eb" candidate_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "122c848014b445728238d1ba49ac509c" is_pre_election: true
I20240502 08:04:27.972728 11560 raft_consensus.cc:2781] T f3666c62e67047bb9393d96859b539eb P 122c848014b445728238d1ba49ac509c [term 1 FOLLOWER]: Leader pre-election won for term 2
W20240502 08:04:27.972787 11288 leader_election.cc:334] T f3666c62e67047bb9393d96859b539eb P 122c848014b445728238d1ba49ac509c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c5e60e7e8697400c8c599205f6db37cf (127.6.204.193:39769): Network error: Client connection negotiation failed: client connection to 127.6.204.193:39769: connect: Connection refused (error 111)
I20240502 08:04:27.973295 11560 raft_consensus.cc:482] T f3666c62e67047bb9393d96859b539eb P 122c848014b445728238d1ba49ac509c [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:04:27.973647 11560 raft_consensus.cc:3037] T f3666c62e67047bb9393d96859b539eb P 122c848014b445728238d1ba49ac509c [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:04:27.978554 11560 raft_consensus.cc:504] T f3666c62e67047bb9393d96859b539eb P 122c848014b445728238d1ba49ac509c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } } peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } }
I20240502 08:04:27.979426 11323 raft_consensus.cc:2379] T f3666c62e67047bb9393d96859b539eb P 122c848014b445728238d1ba49ac509c [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f13c7769fc9f4bfa9796a5c29f715fb4 in current term 2: Already voted for candidate 122c848014b445728238d1ba49ac509c in this term.
I20240502 08:04:27.980264 11560 leader_election.cc:288] T f3666c62e67047bb9393d96859b539eb P 122c848014b445728238d1ba49ac509c [CANDIDATE]: Term 2 election: Requested vote from peers c5e60e7e8697400c8c599205f6db37cf (127.6.204.193:39769), f13c7769fc9f4bfa9796a5c29f715fb4 (127.6.204.195:46495)
I20240502 08:04:27.981998 11398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3666c62e67047bb9393d96859b539eb" candidate_uuid: "122c848014b445728238d1ba49ac509c" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4"
I20240502 08:04:27.982733 11398 raft_consensus.cc:3037] T f3666c62e67047bb9393d96859b539eb P f13c7769fc9f4bfa9796a5c29f715fb4 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:04:27.984215 11364 leader_election.cc:302] T f3666c62e67047bb9393d96859b539eb P f13c7769fc9f4bfa9796a5c29f715fb4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f13c7769fc9f4bfa9796a5c29f715fb4; no voters: 122c848014b445728238d1ba49ac509c, c5e60e7e8697400c8c599205f6db37cf
W20240502 08:04:27.984498 11288 leader_election.cc:334] T f3666c62e67047bb9393d96859b539eb P 122c848014b445728238d1ba49ac509c [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c5e60e7e8697400c8c599205f6db37cf (127.6.204.193:39769): Network error: Client connection negotiation failed: client connection to 127.6.204.193:39769: connect: Connection refused (error 111)
I20240502 08:04:27.988317 11398 raft_consensus.cc:2454] T f3666c62e67047bb9393d96859b539eb P f13c7769fc9f4bfa9796a5c29f715fb4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 122c848014b445728238d1ba49ac509c in term 2.
I20240502 08:04:27.988775 11561 raft_consensus.cc:2726] T f3666c62e67047bb9393d96859b539eb P f13c7769fc9f4bfa9796a5c29f715fb4 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:04:27.989277 11288 leader_election.cc:302] T f3666c62e67047bb9393d96859b539eb P 122c848014b445728238d1ba49ac509c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 122c848014b445728238d1ba49ac509c, f13c7769fc9f4bfa9796a5c29f715fb4; no voters: c5e60e7e8697400c8c599205f6db37cf
I20240502 08:04:27.989982 11560 raft_consensus.cc:2781] T f3666c62e67047bb9393d96859b539eb P 122c848014b445728238d1ba49ac509c [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:04:27.990486 11560 raft_consensus.cc:686] T f3666c62e67047bb9393d96859b539eb P 122c848014b445728238d1ba49ac509c [term 2 LEADER]: Becoming Leader. State: Replica: 122c848014b445728238d1ba49ac509c, State: Running, Role: LEADER
I20240502 08:04:27.991416 11560 consensus_queue.cc:237] T f3666c62e67047bb9393d96859b539eb P 122c848014b445728238d1ba49ac509c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } } peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } }
I20240502 08:04:27.997771 11145 catalog_manager.cc:5484] T f3666c62e67047bb9393d96859b539eb P 122c848014b445728238d1ba49ac509c reported cstate change: term changed from 1 to 2, leader changed from c5e60e7e8697400c8c599205f6db37cf (127.6.204.193) to 122c848014b445728238d1ba49ac509c (127.6.204.194). New cstate: current_term: 2 leader_uuid: "122c848014b445728238d1ba49ac509c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } health_report { overall_health: UNKNOWN } } }
I20240502 08:04:28.002720 11561 raft_consensus.cc:482] T 8c2569709c52401cae6702e5e931ca83 P f13c7769fc9f4bfa9796a5c29f715fb4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:04:28.003158 11561 raft_consensus.cc:504] T 8c2569709c52401cae6702e5e931ca83 P f13c7769fc9f4bfa9796a5c29f715fb4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } } peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } }
I20240502 08:04:28.005156 11561 leader_election.cc:288] T 8c2569709c52401cae6702e5e931ca83 P f13c7769fc9f4bfa9796a5c29f715fb4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5e60e7e8697400c8c599205f6db37cf (127.6.204.193:39769), 122c848014b445728238d1ba49ac509c (127.6.204.194:44945)
I20240502 08:04:28.006285 11323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c2569709c52401cae6702e5e931ca83" candidate_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "122c848014b445728238d1ba49ac509c" is_pre_election: true
I20240502 08:04:28.006942 11323 raft_consensus.cc:2454] T 8c2569709c52401cae6702e5e931ca83 P 122c848014b445728238d1ba49ac509c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f13c7769fc9f4bfa9796a5c29f715fb4 in term 1.
I20240502 08:04:28.007979 11364 leader_election.cc:302] T 8c2569709c52401cae6702e5e931ca83 P f13c7769fc9f4bfa9796a5c29f715fb4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 122c848014b445728238d1ba49ac509c, f13c7769fc9f4bfa9796a5c29f715fb4; no voters:
I20240502 08:04:28.008767 11561 raft_consensus.cc:2781] T 8c2569709c52401cae6702e5e931ca83 P f13c7769fc9f4bfa9796a5c29f715fb4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:04:28.009122 11561 raft_consensus.cc:482] T 8c2569709c52401cae6702e5e931ca83 P f13c7769fc9f4bfa9796a5c29f715fb4 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:04:28.009436 11561 raft_consensus.cc:3037] T 8c2569709c52401cae6702e5e931ca83 P f13c7769fc9f4bfa9796a5c29f715fb4 [term 1 FOLLOWER]: Advancing to term 2
W20240502 08:04:28.009948 11362 leader_election.cc:334] T 8c2569709c52401cae6702e5e931ca83 P f13c7769fc9f4bfa9796a5c29f715fb4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c5e60e7e8697400c8c599205f6db37cf (127.6.204.193:39769): Network error: Client connection negotiation failed: client connection to 127.6.204.193:39769: connect: Connection refused (error 111)
I20240502 08:04:28.014575 11561 raft_consensus.cc:504] T 8c2569709c52401cae6702e5e931ca83 P f13c7769fc9f4bfa9796a5c29f715fb4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } } peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } }
I20240502 08:04:28.016270 11561 leader_election.cc:288] T 8c2569709c52401cae6702e5e931ca83 P f13c7769fc9f4bfa9796a5c29f715fb4 [CANDIDATE]: Term 2 election: Requested vote from peers c5e60e7e8697400c8c599205f6db37cf (127.6.204.193:39769), 122c848014b445728238d1ba49ac509c (127.6.204.194:44945)
I20240502 08:04:28.017555 11323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c2569709c52401cae6702e5e931ca83" candidate_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "122c848014b445728238d1ba49ac509c"
I20240502 08:04:28.018036 11323 raft_consensus.cc:3037] T 8c2569709c52401cae6702e5e931ca83 P 122c848014b445728238d1ba49ac509c [term 1 FOLLOWER]: Advancing to term 2
W20240502 08:04:28.019613 11362 leader_election.cc:334] T 8c2569709c52401cae6702e5e931ca83 P f13c7769fc9f4bfa9796a5c29f715fb4 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c5e60e7e8697400c8c599205f6db37cf (127.6.204.193:39769): Network error: Client connection negotiation failed: client connection to 127.6.204.193:39769: connect: Connection refused (error 111)
I20240502 08:04:28.023376 11323 raft_consensus.cc:2454] T 8c2569709c52401cae6702e5e931ca83 P 122c848014b445728238d1ba49ac509c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f13c7769fc9f4bfa9796a5c29f715fb4 in term 2.
I20240502 08:04:28.024271 11364 leader_election.cc:302] T 8c2569709c52401cae6702e5e931ca83 P f13c7769fc9f4bfa9796a5c29f715fb4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 122c848014b445728238d1ba49ac509c, f13c7769fc9f4bfa9796a5c29f715fb4; no voters: c5e60e7e8697400c8c599205f6db37cf
I20240502 08:04:28.024950 11561 raft_consensus.cc:2781] T 8c2569709c52401cae6702e5e931ca83 P f13c7769fc9f4bfa9796a5c29f715fb4 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:04:28.025954 11561 raft_consensus.cc:686] T 8c2569709c52401cae6702e5e931ca83 P f13c7769fc9f4bfa9796a5c29f715fb4 [term 2 LEADER]: Becoming Leader. State: Replica: f13c7769fc9f4bfa9796a5c29f715fb4, State: Running, Role: LEADER
I20240502 08:04:28.026633 11561 consensus_queue.cc:237] T 8c2569709c52401cae6702e5e931ca83 P f13c7769fc9f4bfa9796a5c29f715fb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } } peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } }
I20240502 08:04:28.032702 11145 catalog_manager.cc:5484] T 8c2569709c52401cae6702e5e931ca83 P f13c7769fc9f4bfa9796a5c29f715fb4 reported cstate change: term changed from 1 to 2, leader changed from c5e60e7e8697400c8c599205f6db37cf (127.6.204.193) to f13c7769fc9f4bfa9796a5c29f715fb4 (127.6.204.195). New cstate: current_term: 2 leader_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } health_report { overall_health: HEALTHY } } }
I20240502 08:04:28.277186 11573 ts_tablet_manager.cc:902] T 743666c9f9a144978e534923cc63222e P f5b0428595be46d891987e38f4a83a15: Initiating tablet copy from peer 122c848014b445728238d1ba49ac509c (127.6.204.194:44945)
I20240502 08:04:28.280126 11573 tablet_copy_client.cc:308] T 743666c9f9a144978e534923cc63222e P f5b0428595be46d891987e38f4a83a15: tablet copy: Beginning tablet copy session from remote peer at address 127.6.204.194:44945
I20240502 08:04:28.289263 11334 tablet_copy_service.cc:140] P 122c848014b445728238d1ba49ac509c: Received BeginTabletCopySession request for tablet 743666c9f9a144978e534923cc63222e from peer f5b0428595be46d891987e38f4a83a15 ({username='slave'} at 127.0.0.1:48392)
I20240502 08:04:28.289731 11334 tablet_copy_service.cc:161] P 122c848014b445728238d1ba49ac509c: Beginning new tablet copy session on tablet 743666c9f9a144978e534923cc63222e from peer f5b0428595be46d891987e38f4a83a15 at {username='slave'} at 127.0.0.1:48392: session id = f5b0428595be46d891987e38f4a83a15-743666c9f9a144978e534923cc63222e
I20240502 08:04:28.295287 11334 tablet_copy_source_session.cc:205] T 743666c9f9a144978e534923cc63222e P 122c848014b445728238d1ba49ac509c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:04:28.297778 11573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 743666c9f9a144978e534923cc63222e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:28.308346 11573 tablet_copy_client.cc:783] T 743666c9f9a144978e534923cc63222e P f5b0428595be46d891987e38f4a83a15: tablet copy: Starting download of 0 data blocks...
I20240502 08:04:28.308815 11573 tablet_copy_client.cc:647] T 743666c9f9a144978e534923cc63222e P f5b0428595be46d891987e38f4a83a15: tablet copy: Starting download of 1 WAL segments...
I20240502 08:04:28.311865 11573 tablet_copy_client.cc:523] T 743666c9f9a144978e534923cc63222e P f5b0428595be46d891987e38f4a83a15: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:04:28.317878 11573 tablet_bootstrap.cc:492] T 743666c9f9a144978e534923cc63222e P f5b0428595be46d891987e38f4a83a15: Bootstrap starting.
I20240502 08:04:28.331072 11573 tablet_bootstrap.cc:492] T 743666c9f9a144978e534923cc63222e P f5b0428595be46d891987e38f4a83a15: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:04:28.331725 11573 tablet_bootstrap.cc:492] T 743666c9f9a144978e534923cc63222e P f5b0428595be46d891987e38f4a83a15: Bootstrap complete.
I20240502 08:04:28.332242 11573 ts_tablet_manager.cc:1367] T 743666c9f9a144978e534923cc63222e P f5b0428595be46d891987e38f4a83a15: Time spent bootstrapping tablet: real 0.015s user 0.012s sys 0.004s
I20240502 08:04:28.334033 11573 raft_consensus.cc:348] T 743666c9f9a144978e534923cc63222e P f5b0428595be46d891987e38f4a83a15 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } } peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } } peers { permanent_uuid: "f5b0428595be46d891987e38f4a83a15" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 36925 } attrs { promote: true } }
I20240502 08:04:28.334673 11573 raft_consensus.cc:729] T 743666c9f9a144978e534923cc63222e P f5b0428595be46d891987e38f4a83a15 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f5b0428595be46d891987e38f4a83a15, State: Initialized, Role: LEARNER
I20240502 08:04:28.335152 11573 consensus_queue.cc:260] T 743666c9f9a144978e534923cc63222e P f5b0428595be46d891987e38f4a83a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } } peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } } peers { permanent_uuid: "f5b0428595be46d891987e38f4a83a15" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 36925 } attrs { promote: true } }
I20240502 08:04:28.337131 11552 heartbeater.cc:503] Master 127.6.204.254:37875 was elected leader, sending a full tablet report...
I20240502 08:04:28.337476 11573 ts_tablet_manager.cc:1398] T 743666c9f9a144978e534923cc63222e P f5b0428595be46d891987e38f4a83a15: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240502 08:04:28.339218 11334 tablet_copy_service.cc:342] P 122c848014b445728238d1ba49ac509c: Request end of tablet copy session f5b0428595be46d891987e38f4a83a15-743666c9f9a144978e534923cc63222e received from {username='slave'} at 127.0.0.1:48392
I20240502 08:04:28.339547 11334 tablet_copy_service.cc:433] P 122c848014b445728238d1ba49ac509c: ending tablet copy session f5b0428595be46d891987e38f4a83a15-743666c9f9a144978e534923cc63222e on tablet 743666c9f9a144978e534923cc63222e with peer f5b0428595be46d891987e38f4a83a15
I20240502 08:04:28.366827 11573 ts_tablet_manager.cc:902] T d117f9452626438cb18dda39df6eb6ed P f5b0428595be46d891987e38f4a83a15: Initiating tablet copy from peer 122c848014b445728238d1ba49ac509c (127.6.204.194:44945)
I20240502 08:04:28.370438 11573 tablet_copy_client.cc:308] T d117f9452626438cb18dda39df6eb6ed P f5b0428595be46d891987e38f4a83a15: tablet copy: Beginning tablet copy session from remote peer at address 127.6.204.194:44945
I20240502 08:04:28.371913 11334 tablet_copy_service.cc:140] P 122c848014b445728238d1ba49ac509c: Received BeginTabletCopySession request for tablet d117f9452626438cb18dda39df6eb6ed from peer f5b0428595be46d891987e38f4a83a15 ({username='slave'} at 127.0.0.1:48392)
I20240502 08:04:28.372310 11334 tablet_copy_service.cc:161] P 122c848014b445728238d1ba49ac509c: Beginning new tablet copy session on tablet d117f9452626438cb18dda39df6eb6ed from peer f5b0428595be46d891987e38f4a83a15 at {username='slave'} at 127.0.0.1:48392: session id = f5b0428595be46d891987e38f4a83a15-d117f9452626438cb18dda39df6eb6ed
I20240502 08:04:28.377971 11334 tablet_copy_source_session.cc:205] T d117f9452626438cb18dda39df6eb6ed P 122c848014b445728238d1ba49ac509c: Tablet Copy: opened 0 blocks and 1 log segments
W20240502 08:04:28.380141 11288 consensus_peers.cc:485] T f3666c62e67047bb9393d96859b539eb P 122c848014b445728238d1ba49ac509c -> Peer c5e60e7e8697400c8c599205f6db37cf (127.6.204.193:39769): Couldn't send request to peer c5e60e7e8697400c8c599205f6db37cf. Status: Network error: Client connection negotiation failed: client connection to 127.6.204.193:39769: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:04:28.381108 11573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d117f9452626438cb18dda39df6eb6ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:28.390973 11573 tablet_copy_client.cc:783] T d117f9452626438cb18dda39df6eb6ed P f5b0428595be46d891987e38f4a83a15: tablet copy: Starting download of 0 data blocks...
I20240502 08:04:28.391454 11573 tablet_copy_client.cc:647] T d117f9452626438cb18dda39df6eb6ed P f5b0428595be46d891987e38f4a83a15: tablet copy: Starting download of 1 WAL segments...
I20240502 08:04:28.394083 11573 tablet_copy_client.cc:523] T d117f9452626438cb18dda39df6eb6ed P f5b0428595be46d891987e38f4a83a15: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:04:28.399596 11573 tablet_bootstrap.cc:492] T d117f9452626438cb18dda39df6eb6ed P f5b0428595be46d891987e38f4a83a15: Bootstrap starting.
I20240502 08:04:28.412910 11573 tablet_bootstrap.cc:492] T d117f9452626438cb18dda39df6eb6ed P f5b0428595be46d891987e38f4a83a15: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:04:28.413509 11573 tablet_bootstrap.cc:492] T d117f9452626438cb18dda39df6eb6ed P f5b0428595be46d891987e38f4a83a15: Bootstrap complete.
I20240502 08:04:28.413986 11573 ts_tablet_manager.cc:1367] T d117f9452626438cb18dda39df6eb6ed P f5b0428595be46d891987e38f4a83a15: Time spent bootstrapping tablet: real 0.015s user 0.011s sys 0.000s
I20240502 08:04:28.415757 11573 raft_consensus.cc:348] T d117f9452626438cb18dda39df6eb6ed P f5b0428595be46d891987e38f4a83a15 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } } peers { permanent_uuid: "f5b0428595be46d891987e38f4a83a15" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 36925 } attrs { promote: true } }
I20240502 08:04:28.416385 11573 raft_consensus.cc:729] T d117f9452626438cb18dda39df6eb6ed P f5b0428595be46d891987e38f4a83a15 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f5b0428595be46d891987e38f4a83a15, State: Initialized, Role: LEARNER
I20240502 08:04:28.416882 11573 consensus_queue.cc:260] T d117f9452626438cb18dda39df6eb6ed P f5b0428595be46d891987e38f4a83a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } } peers { permanent_uuid: "f5b0428595be46d891987e38f4a83a15" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 36925 } attrs { promote: true } }
I20240502 08:04:28.418505 11573 ts_tablet_manager.cc:1398] T d117f9452626438cb18dda39df6eb6ed P f5b0428595be46d891987e38f4a83a15: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240502 08:04:28.420037 11334 tablet_copy_service.cc:342] P 122c848014b445728238d1ba49ac509c: Request end of tablet copy session f5b0428595be46d891987e38f4a83a15-d117f9452626438cb18dda39df6eb6ed received from {username='slave'} at 127.0.0.1:48392
I20240502 08:04:28.420334 11334 tablet_copy_service.cc:433] P 122c848014b445728238d1ba49ac509c: ending tablet copy session f5b0428595be46d891987e38f4a83a15-d117f9452626438cb18dda39df6eb6ed on tablet d117f9452626438cb18dda39df6eb6ed with peer f5b0428595be46d891987e38f4a83a15
I20240502 08:04:28.447898 11323 raft_consensus.cc:1261] T 8c2569709c52401cae6702e5e931ca83 P 122c848014b445728238d1ba49ac509c [term 2 FOLLOWER]: Refusing update from remote peer f13c7769fc9f4bfa9796a5c29f715fb4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240502 08:04:28.449174 11561 consensus_queue.cc:1035] T 8c2569709c52401cae6702e5e931ca83 P f13c7769fc9f4bfa9796a5c29f715fb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:04:28.463024 11362 consensus_peers.cc:485] T 8c2569709c52401cae6702e5e931ca83 P f13c7769fc9f4bfa9796a5c29f715fb4 -> Peer c5e60e7e8697400c8c599205f6db37cf (127.6.204.193:39769): Couldn't send request to peer c5e60e7e8697400c8c599205f6db37cf. Status: Network error: Client connection negotiation failed: client connection to 127.6.204.193:39769: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:04:28.525125 11195 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver c5e60e7e8697400c8c599205f6db37cf not available for placement
I20240502 08:04:28.547930 11398 raft_consensus.cc:1261] T f3666c62e67047bb9393d96859b539eb P f13c7769fc9f4bfa9796a5c29f715fb4 [term 2 FOLLOWER]: Refusing update from remote peer 122c848014b445728238d1ba49ac509c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240502 08:04:28.549291 11560 consensus_queue.cc:1035] T f3666c62e67047bb9393d96859b539eb P 122c848014b445728238d1ba49ac509c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:04:28.779378 11591 consensus_queue.cc:579] T d117f9452626438cb18dda39df6eb6ed P 122c848014b445728238d1ba49ac509c [LEADER]: Leader has been unable to successfully communicate with peer c5e60e7e8697400c8c599205f6db37cf for more than 1 seconds (1.004s)
I20240502 08:04:28.805855 11527 raft_consensus.cc:1203] T d117f9452626438cb18dda39df6eb6ed P f5b0428595be46d891987e38f4a83a15 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:04:28.862915 11527 raft_consensus.cc:1203] T 743666c9f9a144978e534923cc63222e P f5b0428595be46d891987e38f4a83a15 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:04:29.042523 11591 consensus_queue.cc:579] T 743666c9f9a144978e534923cc63222e P 122c848014b445728238d1ba49ac509c [LEADER]: Leader has been unable to successfully communicate with peer c5e60e7e8697400c8c599205f6db37cf for more than 1 seconds (1.353s)
I20240502 08:04:29.174695 11591 consensus_queue.cc:579] T f3666c62e67047bb9393d96859b539eb P 122c848014b445728238d1ba49ac509c [LEADER]: Leader has been unable to successfully communicate with peer c5e60e7e8697400c8c599205f6db37cf for more than 1 seconds (1.182s)
I20240502 08:04:29.183638 11323 consensus_queue.cc:237] T f3666c62e67047bb9393d96859b539eb P 122c848014b445728238d1ba49ac509c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } } peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } } peers { permanent_uuid: "f5b0428595be46d891987e38f4a83a15" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 36925 } attrs { promote: true } }
W20240502 08:04:29.189018 11288 consensus_peers.cc:485] T f3666c62e67047bb9393d96859b539eb P 122c848014b445728238d1ba49ac509c -> Peer f5b0428595be46d891987e38f4a83a15 (127.6.204.196:36925): Couldn't send request to peer f5b0428595be46d891987e38f4a83a15. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f3666c62e67047bb9393d96859b539eb. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:04:29.189210 11398 raft_consensus.cc:1261] T f3666c62e67047bb9393d96859b539eb P f13c7769fc9f4bfa9796a5c29f715fb4 [term 2 FOLLOWER]: Refusing update from remote peer 122c848014b445728238d1ba49ac509c: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:04:29.190547 11592 consensus_queue.cc:1035] T f3666c62e67047bb9393d96859b539eb P 122c848014b445728238d1ba49ac509c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 08:04:29.191865 11288 consensus_peers.cc:485] T f3666c62e67047bb9393d96859b539eb P 122c848014b445728238d1ba49ac509c -> Peer c5e60e7e8697400c8c599205f6db37cf (127.6.204.193:39769): Couldn't send request to peer c5e60e7e8697400c8c599205f6db37cf. Status: Network error: Client connection negotiation failed: client connection to 127.6.204.193:39769: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:04:29.196311 11560 raft_consensus.cc:2932] T f3666c62e67047bb9393d96859b539eb P 122c848014b445728238d1ba49ac509c [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER f5b0428595be46d891987e38f4a83a15 (127.6.204.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } } peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } } peers { permanent_uuid: "f5b0428595be46d891987e38f4a83a15" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 36925 } attrs { promote: true } } }
I20240502 08:04:29.197834 11398 raft_consensus.cc:2932] T f3666c62e67047bb9393d96859b539eb P f13c7769fc9f4bfa9796a5c29f715fb4 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER f5b0428595be46d891987e38f4a83a15 (127.6.204.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } } peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } } peers { permanent_uuid: "f5b0428595be46d891987e38f4a83a15" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 36925 } attrs { promote: true } } }
I20240502 08:04:29.203583 11134 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet f3666c62e67047bb9393d96859b539eb with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 08:04:29.207948 11145 catalog_manager.cc:5484] T f3666c62e67047bb9393d96859b539eb P 122c848014b445728238d1ba49ac509c reported cstate change: config changed from index -1 to 2, NON_VOTER f5b0428595be46d891987e38f4a83a15 (127.6.204.196) added. New cstate: current_term: 2 leader_uuid: "122c848014b445728238d1ba49ac509c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5b0428595be46d891987e38f4a83a15" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 36925 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:04:29.321210 11592 raft_consensus.cc:1050] T 743666c9f9a144978e534923cc63222e P 122c848014b445728238d1ba49ac509c: attempting to promote NON_VOTER f5b0428595be46d891987e38f4a83a15 to VOTER
I20240502 08:04:29.323026 11592 consensus_queue.cc:237] T 743666c9f9a144978e534923cc63222e P 122c848014b445728238d1ba49ac509c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } } peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } } peers { permanent_uuid: "f5b0428595be46d891987e38f4a83a15" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 36925 } attrs { promote: false } }
I20240502 08:04:29.328922 11527 raft_consensus.cc:1261] T 743666c9f9a144978e534923cc63222e P f5b0428595be46d891987e38f4a83a15 [term 1 LEARNER]: Refusing update from remote peer 122c848014b445728238d1ba49ac509c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:04:29.329141 11398 raft_consensus.cc:1261] T 743666c9f9a144978e534923cc63222e P f13c7769fc9f4bfa9796a5c29f715fb4 [term 1 FOLLOWER]: Refusing update from remote peer 122c848014b445728238d1ba49ac509c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:04:29.330536 11592 consensus_queue.cc:1035] T 743666c9f9a144978e534923cc63222e P 122c848014b445728238d1ba49ac509c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:04:29.331246 11588 consensus_queue.cc:579] T 8c2569709c52401cae6702e5e931ca83 P f13c7769fc9f4bfa9796a5c29f715fb4 [LEADER]: Leader has been unable to successfully communicate with peer c5e60e7e8697400c8c599205f6db37cf for more than 1 seconds (1.304s)
I20240502 08:04:29.331269 11598 consensus_queue.cc:1035] T 743666c9f9a144978e534923cc63222e P 122c848014b445728238d1ba49ac509c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5b0428595be46d891987e38f4a83a15" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 36925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20240502 08:04:29.332789 11288 consensus_peers.cc:485] T 743666c9f9a144978e534923cc63222e P 122c848014b445728238d1ba49ac509c -> Peer c5e60e7e8697400c8c599205f6db37cf (127.6.204.193:39769): Couldn't send request to peer c5e60e7e8697400c8c599205f6db37cf. Status: Network error: Client connection negotiation failed: client connection to 127.6.204.193:39769: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:04:29.344776 11397 consensus_queue.cc:237] T 8c2569709c52401cae6702e5e931ca83 P f13c7769fc9f4bfa9796a5c29f715fb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } } peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } } peers { permanent_uuid: "f5b0428595be46d891987e38f4a83a15" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 36925 } attrs { promote: true } }
I20240502 08:04:29.345142 11598 raft_consensus.cc:2932] T 743666c9f9a144978e534923cc63222e P 122c848014b445728238d1ba49ac509c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f5b0428595be46d891987e38f4a83a15 (127.6.204.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } } peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } } peers { permanent_uuid: "f5b0428595be46d891987e38f4a83a15" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 36925 } attrs { promote: false } } }
I20240502 08:04:29.346483 11398 raft_consensus.cc:2932] T 743666c9f9a144978e534923cc63222e P f13c7769fc9f4bfa9796a5c29f715fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f5b0428595be46d891987e38f4a83a15 (127.6.204.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } } peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } } peers { permanent_uuid: "f5b0428595be46d891987e38f4a83a15" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 36925 } attrs { promote: false } } }
I20240502 08:04:29.353899 11323 raft_consensus.cc:1261] T 8c2569709c52401cae6702e5e931ca83 P 122c848014b445728238d1ba49ac509c [term 2 FOLLOWER]: Refusing update from remote peer f13c7769fc9f4bfa9796a5c29f715fb4: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:04:29.355672 11600 consensus_queue.cc:1035] T 8c2569709c52401cae6702e5e931ca83 P f13c7769fc9f4bfa9796a5c29f715fb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:04:29.360602 11146 catalog_manager.cc:5484] T 743666c9f9a144978e534923cc63222e P 122c848014b445728238d1ba49ac509c reported cstate change: config changed from index 2 to 3, f5b0428595be46d891987e38f4a83a15 (127.6.204.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "122c848014b445728238d1ba49ac509c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5b0428595be46d891987e38f4a83a15" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 36925 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:04:29.362382 11588 raft_consensus.cc:2932] T 8c2569709c52401cae6702e5e931ca83 P f13c7769fc9f4bfa9796a5c29f715fb4 [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER f5b0428595be46d891987e38f4a83a15 (127.6.204.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } } peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } } peers { permanent_uuid: "f5b0428595be46d891987e38f4a83a15" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 36925 } attrs { promote: true } } }
I20240502 08:04:29.366551 11323 raft_consensus.cc:2932] T 8c2569709c52401cae6702e5e931ca83 P 122c848014b445728238d1ba49ac509c [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER f5b0428595be46d891987e38f4a83a15 (127.6.204.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } } peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } } peers { permanent_uuid: "f5b0428595be46d891987e38f4a83a15" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 36925 } attrs { promote: true } } }
I20240502 08:04:29.372968 11527 raft_consensus.cc:2932] T 743666c9f9a144978e534923cc63222e P f5b0428595be46d891987e38f4a83a15 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f5b0428595be46d891987e38f4a83a15 (127.6.204.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } } peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } } peers { permanent_uuid: "f5b0428595be46d891987e38f4a83a15" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 36925 } attrs { promote: false } } }
I20240502 08:04:29.379768 11132 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8c2569709c52401cae6702e5e931ca83 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20240502 08:04:29.383478 11362 consensus_peers.cc:485] T 8c2569709c52401cae6702e5e931ca83 P f13c7769fc9f4bfa9796a5c29f715fb4 -> Peer f5b0428595be46d891987e38f4a83a15 (127.6.204.196:36925): Couldn't send request to peer f5b0428595be46d891987e38f4a83a15. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8c2569709c52401cae6702e5e931ca83. This is attempt 1: this message will repeat every 5th retry.
W20240502 08:04:29.384610 11362 consensus_peers.cc:485] T 8c2569709c52401cae6702e5e931ca83 P f13c7769fc9f4bfa9796a5c29f715fb4 -> Peer c5e60e7e8697400c8c599205f6db37cf (127.6.204.193:39769): Couldn't send request to peer c5e60e7e8697400c8c599205f6db37cf. Status: Network error: Client connection negotiation failed: client connection to 127.6.204.193:39769: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:04:29.384908 11145 catalog_manager.cc:5484] T 8c2569709c52401cae6702e5e931ca83 P f13c7769fc9f4bfa9796a5c29f715fb4 reported cstate change: config changed from index -1 to 2, NON_VOTER f5b0428595be46d891987e38f4a83a15 (127.6.204.196) added. New cstate: current_term: 2 leader_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5b0428595be46d891987e38f4a83a15" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 36925 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:04:29.385664 11324 consensus_queue.cc:237] T 743666c9f9a144978e534923cc63222e P 122c848014b445728238d1ba49ac509c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } } peers { permanent_uuid: "f5b0428595be46d891987e38f4a83a15" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 36925 } attrs { promote: false } }
I20240502 08:04:29.395366 11398 raft_consensus.cc:1261] T 743666c9f9a144978e534923cc63222e P f13c7769fc9f4bfa9796a5c29f715fb4 [term 1 FOLLOWER]: Refusing update from remote peer 122c848014b445728238d1ba49ac509c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:04:29.396183 11526 raft_consensus.cc:1261] T 743666c9f9a144978e534923cc63222e P f5b0428595be46d891987e38f4a83a15 [term 1 FOLLOWER]: Refusing update from remote peer 122c848014b445728238d1ba49ac509c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:04:29.396675 11598 consensus_queue.cc:1035] T 743666c9f9a144978e534923cc63222e P 122c848014b445728238d1ba49ac509c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:04:29.399446 11598 consensus_queue.cc:1035] T 743666c9f9a144978e534923cc63222e P 122c848014b445728238d1ba49ac509c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5b0428595be46d891987e38f4a83a15" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 36925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:04:29.405321 11592 raft_consensus.cc:2932] T 743666c9f9a144978e534923cc63222e P 122c848014b445728238d1ba49ac509c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c5e60e7e8697400c8c599205f6db37cf (127.6.204.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } } peers { permanent_uuid: "f5b0428595be46d891987e38f4a83a15" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 36925 } attrs { promote: false } } }
I20240502 08:04:29.407249 11398 raft_consensus.cc:2932] T 743666c9f9a144978e534923cc63222e P f13c7769fc9f4bfa9796a5c29f715fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c5e60e7e8697400c8c599205f6db37cf (127.6.204.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } } peers { permanent_uuid: "f5b0428595be46d891987e38f4a83a15" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 36925 } attrs { promote: false } } }
I20240502 08:04:29.411787 11526 raft_consensus.cc:2932] T 743666c9f9a144978e534923cc63222e P f5b0428595be46d891987e38f4a83a15 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c5e60e7e8697400c8c599205f6db37cf (127.6.204.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } } peers { permanent_uuid: "f5b0428595be46d891987e38f4a83a15" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 36925 } attrs { promote: false } } }
I20240502 08:04:29.415663 11134 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 743666c9f9a144978e534923cc63222e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:04:29.420094 11145 catalog_manager.cc:5484] T 743666c9f9a144978e534923cc63222e P 122c848014b445728238d1ba49ac509c reported cstate change: config changed from index 3 to 4, VOTER c5e60e7e8697400c8c599205f6db37cf (127.6.204.193) evicted. New cstate: current_term: 1 leader_uuid: "122c848014b445728238d1ba49ac509c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5b0428595be46d891987e38f4a83a15" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 36925 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:04:29.425565 11592 raft_consensus.cc:1050] T d117f9452626438cb18dda39df6eb6ed P 122c848014b445728238d1ba49ac509c: attempting to promote NON_VOTER f5b0428595be46d891987e38f4a83a15 to VOTER
I20240502 08:04:29.427307 11592 consensus_queue.cc:237] T d117f9452626438cb18dda39df6eb6ed P 122c848014b445728238d1ba49ac509c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } } peers { permanent_uuid: "f5b0428595be46d891987e38f4a83a15" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 36925 } attrs { promote: false } }
I20240502 08:04:29.435158 11398 raft_consensus.cc:1261] T d117f9452626438cb18dda39df6eb6ed P f13c7769fc9f4bfa9796a5c29f715fb4 [term 1 FOLLOWER]: Refusing update from remote peer 122c848014b445728238d1ba49ac509c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:04:29.435528 11526 raft_consensus.cc:1261] T d117f9452626438cb18dda39df6eb6ed P f5b0428595be46d891987e38f4a83a15 [term 1 LEARNER]: Refusing update from remote peer 122c848014b445728238d1ba49ac509c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20240502 08:04:29.436395 11288 consensus_peers.cc:485] T d117f9452626438cb18dda39df6eb6ed P 122c848014b445728238d1ba49ac509c -> Peer c5e60e7e8697400c8c599205f6db37cf (127.6.204.193:39769): Couldn't send request to peer c5e60e7e8697400c8c599205f6db37cf. Status: Network error: Client connection negotiation failed: client connection to 127.6.204.193:39769: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:04:29.437389 11591 consensus_queue.cc:1035] T d117f9452626438cb18dda39df6eb6ed P 122c848014b445728238d1ba49ac509c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5b0428595be46d891987e38f4a83a15" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 36925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:04:29.437968 11592 consensus_queue.cc:1035] T d117f9452626438cb18dda39df6eb6ed P 122c848014b445728238d1ba49ac509c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240502 08:04:29.450807 11132 catalog_manager.cc:4573] TS c5e60e7e8697400c8c599205f6db37cf (127.6.204.193:39769): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 743666c9f9a144978e534923cc63222e: Network error: Client connection negotiation failed: client connection to 127.6.204.193:39769: connect: Connection refused (error 111)
I20240502 08:04:29.450035 11591 raft_consensus.cc:2932] T d117f9452626438cb18dda39df6eb6ed P 122c848014b445728238d1ba49ac509c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f5b0428595be46d891987e38f4a83a15 (127.6.204.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } } peers { permanent_uuid: "f5b0428595be46d891987e38f4a83a15" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 36925 } attrs { promote: false } } }
I20240502 08:04:29.451870 11526 raft_consensus.cc:2932] T d117f9452626438cb18dda39df6eb6ed P f5b0428595be46d891987e38f4a83a15 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f5b0428595be46d891987e38f4a83a15 (127.6.204.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } } peers { permanent_uuid: "f5b0428595be46d891987e38f4a83a15" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 36925 } attrs { promote: false } } }
I20240502 08:04:29.461345 11398 raft_consensus.cc:2932] T d117f9452626438cb18dda39df6eb6ed P f13c7769fc9f4bfa9796a5c29f715fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f5b0428595be46d891987e38f4a83a15 (127.6.204.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } } peers { permanent_uuid: "f5b0428595be46d891987e38f4a83a15" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 36925 } attrs { promote: false } } }
I20240502 08:04:29.463732 11146 catalog_manager.cc:5484] T d117f9452626438cb18dda39df6eb6ed P 122c848014b445728238d1ba49ac509c reported cstate change: config changed from index 2 to 3, f5b0428595be46d891987e38f4a83a15 (127.6.204.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "122c848014b445728238d1ba49ac509c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5b0428595be46d891987e38f4a83a15" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 36925 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:04:29.475015 11323 consensus_queue.cc:237] T d117f9452626438cb18dda39df6eb6ed P 122c848014b445728238d1ba49ac509c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } } peers { permanent_uuid: "f5b0428595be46d891987e38f4a83a15" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 36925 } attrs { promote: false } }
I20240502 08:04:29.479394 11398 raft_consensus.cc:1261] T d117f9452626438cb18dda39df6eb6ed P f13c7769fc9f4bfa9796a5c29f715fb4 [term 1 FOLLOWER]: Refusing update from remote peer 122c848014b445728238d1ba49ac509c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:04:29.479465 11526 raft_consensus.cc:1261] T d117f9452626438cb18dda39df6eb6ed P f5b0428595be46d891987e38f4a83a15 [term 1 FOLLOWER]: Refusing update from remote peer 122c848014b445728238d1ba49ac509c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:04:29.480806 11591 consensus_queue.cc:1035] T d117f9452626438cb18dda39df6eb6ed P 122c848014b445728238d1ba49ac509c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5b0428595be46d891987e38f4a83a15" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 36925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:04:29.481523 11560 consensus_queue.cc:1035] T d117f9452626438cb18dda39df6eb6ed P 122c848014b445728238d1ba49ac509c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:04:29.487740 11592 raft_consensus.cc:2932] T d117f9452626438cb18dda39df6eb6ed P 122c848014b445728238d1ba49ac509c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c5e60e7e8697400c8c599205f6db37cf (127.6.204.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } } peers { permanent_uuid: "f5b0428595be46d891987e38f4a83a15" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 36925 } attrs { promote: false } } }
I20240502 08:04:29.489925 11526 raft_consensus.cc:2932] T d117f9452626438cb18dda39df6eb6ed P f5b0428595be46d891987e38f4a83a15 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c5e60e7e8697400c8c599205f6db37cf (127.6.204.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } } peers { permanent_uuid: "f5b0428595be46d891987e38f4a83a15" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 36925 } attrs { promote: false } } }
I20240502 08:04:29.489872 11398 raft_consensus.cc:2932] T d117f9452626438cb18dda39df6eb6ed P f13c7769fc9f4bfa9796a5c29f715fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c5e60e7e8697400c8c599205f6db37cf (127.6.204.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } } peers { permanent_uuid: "f5b0428595be46d891987e38f4a83a15" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 36925 } attrs { promote: false } } }
I20240502 08:04:29.498632 11134 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d117f9452626438cb18dda39df6eb6ed with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:04:29.501173 11145 catalog_manager.cc:5484] T d117f9452626438cb18dda39df6eb6ed P 122c848014b445728238d1ba49ac509c reported cstate change: config changed from index 3 to 4, VOTER c5e60e7e8697400c8c599205f6db37cf (127.6.204.193) evicted. New cstate: current_term: 1 leader_uuid: "122c848014b445728238d1ba49ac509c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5b0428595be46d891987e38f4a83a15" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 36925 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 08:04:29.527616 11195 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver c5e60e7e8697400c8c599205f6db37cf not available for placement
I20240502 08:04:29.795593 11611 ts_tablet_manager.cc:902] T f3666c62e67047bb9393d96859b539eb P f5b0428595be46d891987e38f4a83a15: Initiating tablet copy from peer 122c848014b445728238d1ba49ac509c (127.6.204.194:44945)
I20240502 08:04:29.799068 11611 tablet_copy_client.cc:308] T f3666c62e67047bb9393d96859b539eb P f5b0428595be46d891987e38f4a83a15: tablet copy: Beginning tablet copy session from remote peer at address 127.6.204.194:44945
I20240502 08:04:29.800571 11334 tablet_copy_service.cc:140] P 122c848014b445728238d1ba49ac509c: Received BeginTabletCopySession request for tablet f3666c62e67047bb9393d96859b539eb from peer f5b0428595be46d891987e38f4a83a15 ({username='slave'} at 127.0.0.1:48392)
I20240502 08:04:29.800990 11334 tablet_copy_service.cc:161] P 122c848014b445728238d1ba49ac509c: Beginning new tablet copy session on tablet f3666c62e67047bb9393d96859b539eb from peer f5b0428595be46d891987e38f4a83a15 at {username='slave'} at 127.0.0.1:48392: session id = f5b0428595be46d891987e38f4a83a15-f3666c62e67047bb9393d96859b539eb
I20240502 08:04:29.806561 11334 tablet_copy_source_session.cc:205] T f3666c62e67047bb9393d96859b539eb P 122c848014b445728238d1ba49ac509c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:04:29.809010 11611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3666c62e67047bb9393d96859b539eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:29.820259 11611 tablet_copy_client.cc:783] T f3666c62e67047bb9393d96859b539eb P f5b0428595be46d891987e38f4a83a15: tablet copy: Starting download of 0 data blocks...
I20240502 08:04:29.820747 11611 tablet_copy_client.cc:647] T f3666c62e67047bb9393d96859b539eb P f5b0428595be46d891987e38f4a83a15: tablet copy: Starting download of 1 WAL segments...
I20240502 08:04:29.824134 11611 tablet_copy_client.cc:523] T f3666c62e67047bb9393d96859b539eb P f5b0428595be46d891987e38f4a83a15: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:04:29.830112 11611 tablet_bootstrap.cc:492] T f3666c62e67047bb9393d96859b539eb P f5b0428595be46d891987e38f4a83a15: Bootstrap starting.
I20240502 08:04:29.843751 11611 tablet_bootstrap.cc:492] T f3666c62e67047bb9393d96859b539eb P f5b0428595be46d891987e38f4a83a15: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:04:29.844415 11611 tablet_bootstrap.cc:492] T f3666c62e67047bb9393d96859b539eb P f5b0428595be46d891987e38f4a83a15: Bootstrap complete.
I20240502 08:04:29.844944 11611 ts_tablet_manager.cc:1367] T f3666c62e67047bb9393d96859b539eb P f5b0428595be46d891987e38f4a83a15: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.007s
I20240502 08:04:29.846854 11611 raft_consensus.cc:348] T f3666c62e67047bb9393d96859b539eb P f5b0428595be46d891987e38f4a83a15 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } } peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } } peers { permanent_uuid: "f5b0428595be46d891987e38f4a83a15" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 36925 } attrs { promote: true } }
I20240502 08:04:29.847489 11611 raft_consensus.cc:729] T f3666c62e67047bb9393d96859b539eb P f5b0428595be46d891987e38f4a83a15 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: f5b0428595be46d891987e38f4a83a15, State: Initialized, Role: LEARNER
I20240502 08:04:29.847977 11611 consensus_queue.cc:260] T f3666c62e67047bb9393d96859b539eb P f5b0428595be46d891987e38f4a83a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } } peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } } peers { permanent_uuid: "f5b0428595be46d891987e38f4a83a15" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 36925 } attrs { promote: true } }
I20240502 08:04:29.849943 11611 ts_tablet_manager.cc:1398] T f3666c62e67047bb9393d96859b539eb P f5b0428595be46d891987e38f4a83a15: Time spent starting tablet: real 0.005s user 0.003s sys 0.001s
I20240502 08:04:29.851454 11334 tablet_copy_service.cc:342] P 122c848014b445728238d1ba49ac509c: Request end of tablet copy session f5b0428595be46d891987e38f4a83a15-f3666c62e67047bb9393d96859b539eb received from {username='slave'} at 127.0.0.1:48392
I20240502 08:04:29.851831 11334 tablet_copy_service.cc:433] P 122c848014b445728238d1ba49ac509c: ending tablet copy session f5b0428595be46d891987e38f4a83a15-f3666c62e67047bb9393d96859b539eb on tablet f3666c62e67047bb9393d96859b539eb with peer f5b0428595be46d891987e38f4a83a15
I20240502 08:04:29.926580 11611 ts_tablet_manager.cc:902] T 8c2569709c52401cae6702e5e931ca83 P f5b0428595be46d891987e38f4a83a15: Initiating tablet copy from peer f13c7769fc9f4bfa9796a5c29f715fb4 (127.6.204.195:46495)
I20240502 08:04:29.929848 11611 tablet_copy_client.cc:308] T 8c2569709c52401cae6702e5e931ca83 P f5b0428595be46d891987e38f4a83a15: tablet copy: Beginning tablet copy session from remote peer at address 127.6.204.195:46495
I20240502 08:04:29.939118 11408 tablet_copy_service.cc:140] P f13c7769fc9f4bfa9796a5c29f715fb4: Received BeginTabletCopySession request for tablet 8c2569709c52401cae6702e5e931ca83 from peer f5b0428595be46d891987e38f4a83a15 ({username='slave'} at 127.0.0.1:33078)
I20240502 08:04:29.939584 11408 tablet_copy_service.cc:161] P f13c7769fc9f4bfa9796a5c29f715fb4: Beginning new tablet copy session on tablet 8c2569709c52401cae6702e5e931ca83 from peer f5b0428595be46d891987e38f4a83a15 at {username='slave'} at 127.0.0.1:33078: session id = f5b0428595be46d891987e38f4a83a15-8c2569709c52401cae6702e5e931ca83
I20240502 08:04:29.945184 11408 tablet_copy_source_session.cc:205] T 8c2569709c52401cae6702e5e931ca83 P f13c7769fc9f4bfa9796a5c29f715fb4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:04:29.947610 11611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c2569709c52401cae6702e5e931ca83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:29.957500 11611 tablet_copy_client.cc:783] T 8c2569709c52401cae6702e5e931ca83 P f5b0428595be46d891987e38f4a83a15: tablet copy: Starting download of 0 data blocks...
I20240502 08:04:29.957937 11611 tablet_copy_client.cc:647] T 8c2569709c52401cae6702e5e931ca83 P f5b0428595be46d891987e38f4a83a15: tablet copy: Starting download of 1 WAL segments...
I20240502 08:04:29.961081 11611 tablet_copy_client.cc:523] T 8c2569709c52401cae6702e5e931ca83 P f5b0428595be46d891987e38f4a83a15: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:04:29.966732 11611 tablet_bootstrap.cc:492] T 8c2569709c52401cae6702e5e931ca83 P f5b0428595be46d891987e38f4a83a15: Bootstrap starting.
I20240502 08:04:29.979681 11611 tablet_bootstrap.cc:492] T 8c2569709c52401cae6702e5e931ca83 P f5b0428595be46d891987e38f4a83a15: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:04:29.980345 11611 tablet_bootstrap.cc:492] T 8c2569709c52401cae6702e5e931ca83 P f5b0428595be46d891987e38f4a83a15: Bootstrap complete.
I20240502 08:04:29.980861 11611 ts_tablet_manager.cc:1367] T 8c2569709c52401cae6702e5e931ca83 P f5b0428595be46d891987e38f4a83a15: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.002s
I20240502 08:04:29.982631 11611 raft_consensus.cc:348] T 8c2569709c52401cae6702e5e931ca83 P f5b0428595be46d891987e38f4a83a15 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } } peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } } peers { permanent_uuid: "f5b0428595be46d891987e38f4a83a15" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 36925 } attrs { promote: true } }
I20240502 08:04:29.983232 11611 raft_consensus.cc:729] T 8c2569709c52401cae6702e5e931ca83 P f5b0428595be46d891987e38f4a83a15 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: f5b0428595be46d891987e38f4a83a15, State: Initialized, Role: LEARNER
I20240502 08:04:29.983713 11611 consensus_queue.cc:260] T 8c2569709c52401cae6702e5e931ca83 P f5b0428595be46d891987e38f4a83a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } } peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } } peers { permanent_uuid: "f5b0428595be46d891987e38f4a83a15" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 36925 } attrs { promote: true } }
I20240502 08:04:29.985268 11611 ts_tablet_manager.cc:1398] T 8c2569709c52401cae6702e5e931ca83 P f5b0428595be46d891987e38f4a83a15: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240502 08:04:29.986744 11408 tablet_copy_service.cc:342] P f13c7769fc9f4bfa9796a5c29f715fb4: Request end of tablet copy session f5b0428595be46d891987e38f4a83a15-8c2569709c52401cae6702e5e931ca83 received from {username='slave'} at 127.0.0.1:33078
I20240502 08:04:29.987032 11408 tablet_copy_service.cc:433] P f13c7769fc9f4bfa9796a5c29f715fb4: ending tablet copy session f5b0428595be46d891987e38f4a83a15-8c2569709c52401cae6702e5e931ca83 on tablet 8c2569709c52401cae6702e5e931ca83 with peer f5b0428595be46d891987e38f4a83a15
I20240502 08:04:30.182621 11526 raft_consensus.cc:1203] T f3666c62e67047bb9393d96859b539eb P f5b0428595be46d891987e38f4a83a15 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.1->[2.2-2.2] Dedup: 2.2->[]
I20240502 08:04:30.234000 11625 consensus_queue.cc:579] T f3666c62e67047bb9393d96859b539eb P 122c848014b445728238d1ba49ac509c [LEADER]: Leader has been unable to successfully communicate with peer c5e60e7e8697400c8c599205f6db37cf for more than 1 seconds (1.049s)
I20240502 08:04:30.419806 11526 raft_consensus.cc:1203] T 8c2569709c52401cae6702e5e931ca83 P f5b0428595be46d891987e38f4a83a15 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.1->[2.2-2.2] Dedup: 2.2->[]
W20240502 08:04:30.530344 11195 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver c5e60e7e8697400c8c599205f6db37cf not available for placement
I20240502 08:04:30.641602 11625 raft_consensus.cc:1050] T f3666c62e67047bb9393d96859b539eb P 122c848014b445728238d1ba49ac509c: attempting to promote NON_VOTER f5b0428595be46d891987e38f4a83a15 to VOTER
I20240502 08:04:30.643481 11625 consensus_queue.cc:237] T f3666c62e67047bb9393d96859b539eb P 122c848014b445728238d1ba49ac509c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 0, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } } peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } } peers { permanent_uuid: "f5b0428595be46d891987e38f4a83a15" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 36925 } attrs { promote: false } }
I20240502 08:04:30.649169 11526 raft_consensus.cc:1261] T f3666c62e67047bb9393d96859b539eb P f5b0428595be46d891987e38f4a83a15 [term 2 LEARNER]: Refusing update from remote peer 122c848014b445728238d1ba49ac509c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 08:04:30.649847 11398 raft_consensus.cc:1261] T f3666c62e67047bb9393d96859b539eb P f13c7769fc9f4bfa9796a5c29f715fb4 [term 2 FOLLOWER]: Refusing update from remote peer 122c848014b445728238d1ba49ac509c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 08:04:30.651367 11631 consensus_queue.cc:1035] T f3666c62e67047bb9393d96859b539eb P 122c848014b445728238d1ba49ac509c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5b0428595be46d891987e38f4a83a15" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 36925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:04:30.652308 11625 consensus_queue.cc:1035] T f3666c62e67047bb9393d96859b539eb P 122c848014b445728238d1ba49ac509c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240502 08:04:30.653071 11288 consensus_peers.cc:485] T f3666c62e67047bb9393d96859b539eb P 122c848014b445728238d1ba49ac509c -> Peer c5e60e7e8697400c8c599205f6db37cf (127.6.204.193:39769): Couldn't send request to peer c5e60e7e8697400c8c599205f6db37cf. Status: Network error: Client connection negotiation failed: client connection to 127.6.204.193:39769: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:04:30.660143 11631 raft_consensus.cc:2932] T f3666c62e67047bb9393d96859b539eb P 122c848014b445728238d1ba49ac509c [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index 2 to 3, f5b0428595be46d891987e38f4a83a15 (127.6.204.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } } peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } } peers { permanent_uuid: "f5b0428595be46d891987e38f4a83a15" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 36925 } attrs { promote: false } } }
I20240502 08:04:30.661753 11526 raft_consensus.cc:2932] T f3666c62e67047bb9393d96859b539eb P f5b0428595be46d891987e38f4a83a15 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, f5b0428595be46d891987e38f4a83a15 (127.6.204.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } } peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } } peers { permanent_uuid: "f5b0428595be46d891987e38f4a83a15" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 36925 } attrs { promote: false } } }
I20240502 08:04:30.671572 11398 raft_consensus.cc:2932] T f3666c62e67047bb9393d96859b539eb P f13c7769fc9f4bfa9796a5c29f715fb4 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, f5b0428595be46d891987e38f4a83a15 (127.6.204.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } } peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } } peers { permanent_uuid: "f5b0428595be46d891987e38f4a83a15" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 36925 } attrs { promote: false } } }
I20240502 08:04:30.674015 11147 catalog_manager.cc:5484] T f3666c62e67047bb9393d96859b539eb P 122c848014b445728238d1ba49ac509c reported cstate change: config changed from index 2 to 3, f5b0428595be46d891987e38f4a83a15 (127.6.204.196) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "122c848014b445728238d1ba49ac509c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5b0428595be46d891987e38f4a83a15" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 36925 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 08:04:30.680961 11132 catalog_manager.cc:4573] TS c5e60e7e8697400c8c599205f6db37cf (127.6.204.193:39769): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 743666c9f9a144978e534923cc63222e: Network error: Client connection negotiation failed: client connection to 127.6.204.193:39769: connect: Connection refused (error 111)
I20240502 08:04:30.691326 11323 consensus_queue.cc:237] T f3666c62e67047bb9393d96859b539eb P 122c848014b445728238d1ba49ac509c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } } peers { permanent_uuid: "f5b0428595be46d891987e38f4a83a15" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 36925 } attrs { promote: false } }
I20240502 08:04:30.696624 11526 raft_consensus.cc:1261] T f3666c62e67047bb9393d96859b539eb P f5b0428595be46d891987e38f4a83a15 [term 2 FOLLOWER]: Refusing update from remote peer 122c848014b445728238d1ba49ac509c: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 08:04:30.697381 11398 raft_consensus.cc:1261] T f3666c62e67047bb9393d96859b539eb P f13c7769fc9f4bfa9796a5c29f715fb4 [term 2 FOLLOWER]: Refusing update from remote peer 122c848014b445728238d1ba49ac509c: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 08:04:30.698118 11560 consensus_queue.cc:1035] T f3666c62e67047bb9393d96859b539eb P 122c848014b445728238d1ba49ac509c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5b0428595be46d891987e38f4a83a15" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 36925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:04:30.698866 11624 consensus_queue.cc:1035] T f3666c62e67047bb9393d96859b539eb P 122c848014b445728238d1ba49ac509c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:04:30.704911 11625 raft_consensus.cc:2932] T f3666c62e67047bb9393d96859b539eb P 122c848014b445728238d1ba49ac509c [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER c5e60e7e8697400c8c599205f6db37cf (127.6.204.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } } peers { permanent_uuid: "f5b0428595be46d891987e38f4a83a15" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 36925 } attrs { promote: false } } }
I20240502 08:04:30.706718 11526 raft_consensus.cc:2932] T f3666c62e67047bb9393d96859b539eb P f5b0428595be46d891987e38f4a83a15 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER c5e60e7e8697400c8c599205f6db37cf (127.6.204.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } } peers { permanent_uuid: "f5b0428595be46d891987e38f4a83a15" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 36925 } attrs { promote: false } } }
I20240502 08:04:30.706718 11398 raft_consensus.cc:2932] T f3666c62e67047bb9393d96859b539eb P f13c7769fc9f4bfa9796a5c29f715fb4 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER c5e60e7e8697400c8c599205f6db37cf (127.6.204.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } } peers { permanent_uuid: "f5b0428595be46d891987e38f4a83a15" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 36925 } attrs { promote: false } } }
I20240502 08:04:30.712209 11134 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f3666c62e67047bb9393d96859b539eb with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:04:30.715616 11147 catalog_manager.cc:5484] T f3666c62e67047bb9393d96859b539eb P 122c848014b445728238d1ba49ac509c reported cstate change: config changed from index 3 to 4, VOTER c5e60e7e8697400c8c599205f6db37cf (127.6.204.193) evicted. New cstate: current_term: 2 leader_uuid: "122c848014b445728238d1ba49ac509c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5b0428595be46d891987e38f4a83a15" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 36925 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:04:30.837704 11629 consensus_queue.cc:579] T 8c2569709c52401cae6702e5e931ca83 P f13c7769fc9f4bfa9796a5c29f715fb4 [LEADER]: Leader has been unable to successfully communicate with peer c5e60e7e8697400c8c599205f6db37cf for more than 1 seconds (1.491s)
I20240502 08:04:30.973242 11629 raft_consensus.cc:1050] T 8c2569709c52401cae6702e5e931ca83 P f13c7769fc9f4bfa9796a5c29f715fb4: attempting to promote NON_VOTER f5b0428595be46d891987e38f4a83a15 to VOTER
I20240502 08:04:30.975332 11629 consensus_queue.cc:237] T 8c2569709c52401cae6702e5e931ca83 P f13c7769fc9f4bfa9796a5c29f715fb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 0, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } } peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } } peers { permanent_uuid: "f5b0428595be46d891987e38f4a83a15" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 36925 } attrs { promote: false } }
I20240502 08:04:30.981176 11323 raft_consensus.cc:1261] T 8c2569709c52401cae6702e5e931ca83 P 122c848014b445728238d1ba49ac509c [term 2 FOLLOWER]: Refusing update from remote peer f13c7769fc9f4bfa9796a5c29f715fb4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 08:04:30.981818 11526 raft_consensus.cc:1261] T 8c2569709c52401cae6702e5e931ca83 P f5b0428595be46d891987e38f4a83a15 [term 2 LEARNER]: Refusing update from remote peer f13c7769fc9f4bfa9796a5c29f715fb4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 08:04:30.982816 11643 consensus_queue.cc:1035] T 8c2569709c52401cae6702e5e931ca83 P f13c7769fc9f4bfa9796a5c29f715fb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 08:04:30.983877 11629 consensus_queue.cc:1035] T 8c2569709c52401cae6702e5e931ca83 P f13c7769fc9f4bfa9796a5c29f715fb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5b0428595be46d891987e38f4a83a15" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 36925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240502 08:04:30.984760 11362 consensus_peers.cc:485] T 8c2569709c52401cae6702e5e931ca83 P f13c7769fc9f4bfa9796a5c29f715fb4 -> Peer c5e60e7e8697400c8c599205f6db37cf (127.6.204.193:39769): Couldn't send request to peer c5e60e7e8697400c8c599205f6db37cf. Status: Network error: Client connection negotiation failed: client connection to 127.6.204.193:39769: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:04:30.990967 11643 raft_consensus.cc:2932] T 8c2569709c52401cae6702e5e931ca83 P f13c7769fc9f4bfa9796a5c29f715fb4 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index 2 to 3, f5b0428595be46d891987e38f4a83a15 (127.6.204.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } } peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } } peers { permanent_uuid: "f5b0428595be46d891987e38f4a83a15" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 36925 } attrs { promote: false } } }
I20240502 08:04:30.992537 11526 raft_consensus.cc:2932] T 8c2569709c52401cae6702e5e931ca83 P f5b0428595be46d891987e38f4a83a15 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, f5b0428595be46d891987e38f4a83a15 (127.6.204.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } } peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } } peers { permanent_uuid: "f5b0428595be46d891987e38f4a83a15" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 36925 } attrs { promote: false } } }
I20240502 08:04:30.999872 11323 raft_consensus.cc:2932] T 8c2569709c52401cae6702e5e931ca83 P 122c848014b445728238d1ba49ac509c [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, f5b0428595be46d891987e38f4a83a15 (127.6.204.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } } peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } } peers { permanent_uuid: "f5b0428595be46d891987e38f4a83a15" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 36925 } attrs { promote: false } } }
I20240502 08:04:31.002616 11147 catalog_manager.cc:5484] T 8c2569709c52401cae6702e5e931ca83 P f13c7769fc9f4bfa9796a5c29f715fb4 reported cstate change: config changed from index 2 to 3, f5b0428595be46d891987e38f4a83a15 (127.6.204.196) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c5e60e7e8697400c8c599205f6db37cf" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 39769 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5b0428595be46d891987e38f4a83a15" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 36925 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:04:31.015945 11398 consensus_queue.cc:237] T 8c2569709c52401cae6702e5e931ca83 P f13c7769fc9f4bfa9796a5c29f715fb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } } peers { permanent_uuid: "f5b0428595be46d891987e38f4a83a15" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 36925 } attrs { promote: false } }
I20240502 08:04:31.020817 11323 raft_consensus.cc:1261] T 8c2569709c52401cae6702e5e931ca83 P 122c848014b445728238d1ba49ac509c [term 2 FOLLOWER]: Refusing update from remote peer f13c7769fc9f4bfa9796a5c29f715fb4: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 08:04:31.022084 11629 consensus_queue.cc:1035] T 8c2569709c52401cae6702e5e931ca83 P f13c7769fc9f4bfa9796a5c29f715fb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:04:31.022907 11526 raft_consensus.cc:1261] T 8c2569709c52401cae6702e5e931ca83 P f5b0428595be46d891987e38f4a83a15 [term 2 FOLLOWER]: Refusing update from remote peer f13c7769fc9f4bfa9796a5c29f715fb4: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 08:04:31.024305 11629 consensus_queue.cc:1035] T 8c2569709c52401cae6702e5e931ca83 P f13c7769fc9f4bfa9796a5c29f715fb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5b0428595be46d891987e38f4a83a15" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 36925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:04:31.028196 11628 raft_consensus.cc:2932] T 8c2569709c52401cae6702e5e931ca83 P f13c7769fc9f4bfa9796a5c29f715fb4 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER c5e60e7e8697400c8c599205f6db37cf (127.6.204.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } } peers { permanent_uuid: "f5b0428595be46d891987e38f4a83a15" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 36925 } attrs { promote: false } } }
I20240502 08:04:31.029754 11323 raft_consensus.cc:2932] T 8c2569709c52401cae6702e5e931ca83 P 122c848014b445728238d1ba49ac509c [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER c5e60e7e8697400c8c599205f6db37cf (127.6.204.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } } peers { permanent_uuid: "f5b0428595be46d891987e38f4a83a15" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 36925 } attrs { promote: false } } }
I20240502 08:04:31.032426 11526 raft_consensus.cc:2932] T 8c2569709c52401cae6702e5e931ca83 P f5b0428595be46d891987e38f4a83a15 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER c5e60e7e8697400c8c599205f6db37cf (127.6.204.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } } peers { permanent_uuid: "f5b0428595be46d891987e38f4a83a15" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 36925 } attrs { promote: false } } }
I20240502 08:04:31.040881 11132 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8c2569709c52401cae6702e5e931ca83 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:04:31.042083 11147 catalog_manager.cc:5484] T 8c2569709c52401cae6702e5e931ca83 P f13c7769fc9f4bfa9796a5c29f715fb4 reported cstate change: config changed from index 3 to 4, VOTER c5e60e7e8697400c8c599205f6db37cf (127.6.204.193) evicted. New cstate: current_term: 2 leader_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "122c848014b445728238d1ba49ac509c" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 44945 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f13c7769fc9f4bfa9796a5c29f715fb4" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5b0428595be46d891987e38f4a83a15" member_type: VOTER last_known_addr { host: "127.6.204.196" port: 36925 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 08:04:31.689925 11132 catalog_manager.cc:4573] TS c5e60e7e8697400c8c599205f6db37cf (127.6.204.193:39769): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 8c2569709c52401cae6702e5e931ca83: Network error: Client connection negotiation failed: client connection to 127.6.204.193:39769: connect: Connection refused (error 111)
W20240502 08:04:31.959606 11132 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.204.193:39769: connect: Connection refused (error 111) [suppressed 63 similar messages]
W20240502 08:04:33.038393 11132 catalog_manager.cc:4573] TS c5e60e7e8697400c8c599205f6db37cf (127.6.204.193:39769): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet f3666c62e67047bb9393d96859b539eb: Network error: Client connection negotiation failed: client connection to 127.6.204.193:39769: connect: Connection refused (error 111)
W20240502 08:04:34.462108 11178 debug-util.cc:398] Leaking SignalData structure 0x7b08001ca300 after lost signal to thread 6966
W20240502 08:04:34.463860 11178 debug-util.cc:398] Leaking SignalData structure 0x7b080023ab80 after lost signal to thread 11181
W20240502 08:04:34.465366 11178 debug-util.cc:398] Leaking SignalData structure 0x7b080002f4e0 after lost signal to thread 11348
W20240502 08:04:34.466676 11178 debug-util.cc:398] Leaking SignalData structure 0x7b0800240e00 after lost signal to thread 11422
W20240502 08:04:34.467998 11178 debug-util.cc:398] Leaking SignalData structure 0x7b08001d1f40 after lost signal to thread 11551
I20240502 08:04:34.666610 6963 tablet_server.cc:178] TabletServer@127.6.204.194:0 shutting down...
I20240502 08:04:34.711534 6963 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 08:04:34.712430 6963 tablet_replica.cc:330] T 743666c9f9a144978e534923cc63222e P 122c848014b445728238d1ba49ac509c: stopping tablet replica
I20240502 08:04:34.713383 6963 raft_consensus.cc:2229] T 743666c9f9a144978e534923cc63222e P 122c848014b445728238d1ba49ac509c [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:04:34.714818 6963 raft_consensus.cc:2258] T 743666c9f9a144978e534923cc63222e P 122c848014b445728238d1ba49ac509c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:04:34.718079 6963 tablet_replica.cc:330] T 8c2569709c52401cae6702e5e931ca83 P 122c848014b445728238d1ba49ac509c: stopping tablet replica
I20240502 08:04:34.718942 6963 raft_consensus.cc:2229] T 8c2569709c52401cae6702e5e931ca83 P 122c848014b445728238d1ba49ac509c [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:04:34.721292 6963 raft_consensus.cc:2258] T 8c2569709c52401cae6702e5e931ca83 P 122c848014b445728238d1ba49ac509c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:04:34.725503 6963 tablet_replica.cc:330] T f3666c62e67047bb9393d96859b539eb P 122c848014b445728238d1ba49ac509c: stopping tablet replica
I20240502 08:04:34.726519 6963 raft_consensus.cc:2229] T f3666c62e67047bb9393d96859b539eb P 122c848014b445728238d1ba49ac509c [term 2 LEADER]: Raft consensus shutting down.
I20240502 08:04:34.727834 6963 raft_consensus.cc:2258] T f3666c62e67047bb9393d96859b539eb P 122c848014b445728238d1ba49ac509c [term 2 FOLLOWER]: Raft consensus is shut down!
W20240502 08:04:34.729297 11364 consensus_peers.cc:485] T 8c2569709c52401cae6702e5e931ca83 P f13c7769fc9f4bfa9796a5c29f715fb4 -> Peer 122c848014b445728238d1ba49ac509c (127.6.204.194:44945): Couldn't send request to peer 122c848014b445728238d1ba49ac509c. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:04:34.730790 6963 tablet_replica.cc:330] T d117f9452626438cb18dda39df6eb6ed P 122c848014b445728238d1ba49ac509c: stopping tablet replica
I20240502 08:04:34.731375 6963 raft_consensus.cc:2229] T d117f9452626438cb18dda39df6eb6ed P 122c848014b445728238d1ba49ac509c [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:04:34.732595 6963 raft_consensus.cc:2258] T d117f9452626438cb18dda39df6eb6ed P 122c848014b445728238d1ba49ac509c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:04:34.760506 6963 tablet_server.cc:194] TabletServer@127.6.204.194:0 shutdown complete.
I20240502 08:04:34.781389 6963 tablet_server.cc:178] TabletServer@127.6.204.195:0 shutting down...
I20240502 08:04:34.828270 6963 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 08:04:34.828997 6963 tablet_replica.cc:330] T 743666c9f9a144978e534923cc63222e P f13c7769fc9f4bfa9796a5c29f715fb4: stopping tablet replica
I20240502 08:04:34.829681 6963 raft_consensus.cc:2229] T 743666c9f9a144978e534923cc63222e P f13c7769fc9f4bfa9796a5c29f715fb4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:04:34.831112 6963 raft_consensus.cc:2258] T 743666c9f9a144978e534923cc63222e P f13c7769fc9f4bfa9796a5c29f715fb4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:04:34.833688 6963 tablet_replica.cc:330] T d117f9452626438cb18dda39df6eb6ed P f13c7769fc9f4bfa9796a5c29f715fb4: stopping tablet replica
I20240502 08:04:34.834493 6963 raft_consensus.cc:2229] T d117f9452626438cb18dda39df6eb6ed P f13c7769fc9f4bfa9796a5c29f715fb4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:04:34.835763 6963 raft_consensus.cc:2258] T d117f9452626438cb18dda39df6eb6ed P f13c7769fc9f4bfa9796a5c29f715fb4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:04:34.838045 6963 tablet_replica.cc:330] T 8c2569709c52401cae6702e5e931ca83 P f13c7769fc9f4bfa9796a5c29f715fb4: stopping tablet replica
I20240502 08:04:34.838660 6963 raft_consensus.cc:2229] T 8c2569709c52401cae6702e5e931ca83 P f13c7769fc9f4bfa9796a5c29f715fb4 [term 2 LEADER]: Raft consensus shutting down.
I20240502 08:04:34.839756 6963 raft_consensus.cc:2258] T 8c2569709c52401cae6702e5e931ca83 P f13c7769fc9f4bfa9796a5c29f715fb4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:04:34.841975 6963 tablet_replica.cc:330] T f3666c62e67047bb9393d96859b539eb P f13c7769fc9f4bfa9796a5c29f715fb4: stopping tablet replica
I20240502 08:04:34.842548 6963 raft_consensus.cc:2229] T f3666c62e67047bb9393d96859b539eb P f13c7769fc9f4bfa9796a5c29f715fb4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:04:34.843737 6963 raft_consensus.cc:2258] T f3666c62e67047bb9393d96859b539eb P f13c7769fc9f4bfa9796a5c29f715fb4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:04:34.873631 6963 tablet_server.cc:194] TabletServer@127.6.204.195:0 shutdown complete.
I20240502 08:04:34.893575 6963 tablet_server.cc:178] TabletServer@127.6.204.196:0 shutting down...
I20240502 08:04:34.939152 6963 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 08:04:34.939975 6963 tablet_replica.cc:330] T 8c2569709c52401cae6702e5e931ca83 P f5b0428595be46d891987e38f4a83a15: stopping tablet replica
I20240502 08:04:34.940783 6963 raft_consensus.cc:2229] T 8c2569709c52401cae6702e5e931ca83 P f5b0428595be46d891987e38f4a83a15 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:04:34.941679 6963 raft_consensus.cc:2258] T 8c2569709c52401cae6702e5e931ca83 P f5b0428595be46d891987e38f4a83a15 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:04:34.944361 6963 tablet_replica.cc:330] T f3666c62e67047bb9393d96859b539eb P f5b0428595be46d891987e38f4a83a15: stopping tablet replica
I20240502 08:04:34.944906 6963 raft_consensus.cc:2229] T f3666c62e67047bb9393d96859b539eb P f5b0428595be46d891987e38f4a83a15 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:04:34.945650 6963 raft_consensus.cc:2258] T f3666c62e67047bb9393d96859b539eb P f5b0428595be46d891987e38f4a83a15 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:04:34.947858 6963 tablet_replica.cc:330] T d117f9452626438cb18dda39df6eb6ed P f5b0428595be46d891987e38f4a83a15: stopping tablet replica
I20240502 08:04:34.948383 6963 raft_consensus.cc:2229] T d117f9452626438cb18dda39df6eb6ed P f5b0428595be46d891987e38f4a83a15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:04:34.949221 6963 raft_consensus.cc:2258] T d117f9452626438cb18dda39df6eb6ed P f5b0428595be46d891987e38f4a83a15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:04:34.951748 6963 tablet_replica.cc:330] T 743666c9f9a144978e534923cc63222e P f5b0428595be46d891987e38f4a83a15: stopping tablet replica
I20240502 08:04:34.952258 6963 raft_consensus.cc:2229] T 743666c9f9a144978e534923cc63222e P f5b0428595be46d891987e38f4a83a15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:04:34.953166 6963 raft_consensus.cc:2258] T 743666c9f9a144978e534923cc63222e P f5b0428595be46d891987e38f4a83a15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:04:34.979262 6963 tablet_server.cc:194] TabletServer@127.6.204.196:0 shutdown complete.
I20240502 08:04:35.004060 6963 master.cc:537] Master@127.6.204.254:37875 shutting down...
W20240502 08:04:35.142374 11132 catalog_manager.cc:4573] TS c5e60e7e8697400c8c599205f6db37cf (127.6.204.193:39769): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet f3666c62e67047bb9393d96859b539eb: Network error: Client connection negotiation failed: client connection to 127.6.204.193:39769: connect: Connection refused (error 111)
W20240502 08:04:37.939322 11132 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.204.193:39769: connect: Connection refused (error 111) [suppressed 8 similar messages]
W20240502 08:04:37.942646 11132 catalog_manager.cc:4573] TS c5e60e7e8697400c8c599205f6db37cf (127.6.204.193:39769): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 743666c9f9a144978e534923cc63222e: Network error: Client connection negotiation failed: client connection to 127.6.204.193:39769: connect: Connection refused (error 111)
I20240502 08:04:38.627593 6963 raft_consensus.cc:2229] T 00000000000000000000000000000000 P c6e176cf9790442c95418718e7c71e08 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:04:38.629204 6963 raft_consensus.cc:2258] T 00000000000000000000000000000000 P c6e176cf9790442c95418718e7c71e08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:04:38.629508 6963 tablet_replica.cc:330] T 00000000000000000000000000000000 P c6e176cf9790442c95418718e7c71e08: stopping tablet replica
I20240502 08:04:38.668170 6963 master.cc:552] Master@127.6.204.254:37875 shutdown complete.
[ OK ] AutoRebalancerTest.NoRebalancingIfReplicasRecovering (14316 ms)
[ RUN ] AutoRebalancerTest.TestHandlingFailedTservers
I20240502 08:04:38.708235 6963 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.6.204.254:38291
I20240502 08:04:38.709302 6963 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:04:38.715610 11659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:04:38.715798 11660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:04:38.716502 11662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:04:38.716868 6963 server_base.cc:1034] running on GCE node
I20240502 08:04:38.717712 6963 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:04:38.717895 6963 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:04:38.718009 6963 hybrid_clock.cc:648] HybridClock initialized: now 1714637078717998 us; error 0 us; skew 500 ppm
I20240502 08:04:38.718490 6963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:04:38.726522 6963 webserver.cc:460] Webserver started at http://127.6.204.254:42179/ using document root <none> and password file <none>
I20240502 08:04:38.726989 6963 fs_manager.cc:359] Metadata directory not provided
I20240502 08:04:38.727141 6963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:04:38.727370 6963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:04:38.728554 6963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714636933625983-6963-0/minicluster-data/master-0-root/instance:
uuid: "7b1d8ba2ea8b49f7a7b78e8019c8c0bf"
format_stamp: "Formatted at 2024-05-02 08:04:38 on dist-test-slave-32xs"
I20240502 08:04:38.733086 6963 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240502 08:04:38.736690 11667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:04:38.737514 6963 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 08:04:38.737756 6963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714636933625983-6963-0/minicluster-data/master-0-root
uuid: "7b1d8ba2ea8b49f7a7b78e8019c8c0bf"
format_stamp: "Formatted at 2024-05-02 08:04:38 on dist-test-slave-32xs"
I20240502 08:04:38.737998 6963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714636933625983-6963-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714636933625983-6963-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714636933625983-6963-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:04:38.754074 6963 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:04:38.755306 6963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:04:38.795087 6963 rpc_server.cc:307] RPC server started. Bound to: 127.6.204.254:38291
I20240502 08:04:38.795181 11718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.204.254:38291 every 8 connection(s)
I20240502 08:04:38.799044 11719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:38.810618 11719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b1d8ba2ea8b49f7a7b78e8019c8c0bf: Bootstrap starting.
I20240502 08:04:38.815106 11719 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b1d8ba2ea8b49f7a7b78e8019c8c0bf: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:38.819447 11719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b1d8ba2ea8b49f7a7b78e8019c8c0bf: No bootstrap required, opened a new log
I20240502 08:04:38.821552 11719 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b1d8ba2ea8b49f7a7b78e8019c8c0bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b1d8ba2ea8b49f7a7b78e8019c8c0bf" member_type: VOTER }
I20240502 08:04:38.822016 11719 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b1d8ba2ea8b49f7a7b78e8019c8c0bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:38.822300 11719 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b1d8ba2ea8b49f7a7b78e8019c8c0bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b1d8ba2ea8b49f7a7b78e8019c8c0bf, State: Initialized, Role: FOLLOWER
I20240502 08:04:38.822897 11719 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b1d8ba2ea8b49f7a7b78e8019c8c0bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b1d8ba2ea8b49f7a7b78e8019c8c0bf" member_type: VOTER }
I20240502 08:04:38.823370 11719 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b1d8ba2ea8b49f7a7b78e8019c8c0bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:04:38.823580 11719 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b1d8ba2ea8b49f7a7b78e8019c8c0bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:04:38.823841 11719 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b1d8ba2ea8b49f7a7b78e8019c8c0bf [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:38.828569 11719 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b1d8ba2ea8b49f7a7b78e8019c8c0bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b1d8ba2ea8b49f7a7b78e8019c8c0bf" member_type: VOTER }
I20240502 08:04:38.829092 11719 leader_election.cc:302] T 00000000000000000000000000000000 P 7b1d8ba2ea8b49f7a7b78e8019c8c0bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b1d8ba2ea8b49f7a7b78e8019c8c0bf; no voters:
I20240502 08:04:38.830336 11719 leader_election.cc:288] T 00000000000000000000000000000000 P 7b1d8ba2ea8b49f7a7b78e8019c8c0bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:04:38.830659 11722 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b1d8ba2ea8b49f7a7b78e8019c8c0bf [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:04:38.831982 11722 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b1d8ba2ea8b49f7a7b78e8019c8c0bf [term 1 LEADER]: Becoming Leader. State: Replica: 7b1d8ba2ea8b49f7a7b78e8019c8c0bf, State: Running, Role: LEADER
I20240502 08:04:38.832715 11722 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b1d8ba2ea8b49f7a7b78e8019c8c0bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b1d8ba2ea8b49f7a7b78e8019c8c0bf" member_type: VOTER }
I20240502 08:04:38.833276 11719 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b1d8ba2ea8b49f7a7b78e8019c8c0bf [sys.catalog]: configured and running, proceeding with master startup.
I20240502 08:04:38.837323 11723 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b1d8ba2ea8b49f7a7b78e8019c8c0bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b1d8ba2ea8b49f7a7b78e8019c8c0bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b1d8ba2ea8b49f7a7b78e8019c8c0bf" member_type: VOTER } }
I20240502 08:04:38.837675 11724 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b1d8ba2ea8b49f7a7b78e8019c8c0bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b1d8ba2ea8b49f7a7b78e8019c8c0bf. Latest consensus state: current_term: 1 leader_uuid: "7b1d8ba2ea8b49f7a7b78e8019c8c0bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b1d8ba2ea8b49f7a7b78e8019c8c0bf" member_type: VOTER } }
I20240502 08:04:38.838253 11723 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b1d8ba2ea8b49f7a7b78e8019c8c0bf [sys.catalog]: This master's current role is: LEADER
I20240502 08:04:38.838622 11724 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b1d8ba2ea8b49f7a7b78e8019c8c0bf [sys.catalog]: This master's current role is: LEADER
I20240502 08:04:38.841547 11728 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 08:04:38.849763 11728 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 08:04:38.851212 6963 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 08:04:38.859218 11728 catalog_manager.cc:1347] Generated new cluster ID: be4133decf28411384c068755ce139ff
I20240502 08:04:38.859789 11728 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 08:04:38.918395 11728 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 08:04:38.919735 11728 catalog_manager.cc:1504] Loading token signing keys...
I20240502 08:04:38.937608 11728 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 7b1d8ba2ea8b49f7a7b78e8019c8c0bf: Generated new TSK 0
I20240502 08:04:38.938330 11728 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 08:04:38.982342 6963 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:04:38.988422 11740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:04:38.989301 11741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:04:38.990783 11743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:04:38.991449 6963 server_base.cc:1034] running on GCE node
I20240502 08:04:38.992213 6963 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:04:38.992390 6963 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:04:38.992512 6963 hybrid_clock.cc:648] HybridClock initialized: now 1714637078992501 us; error 0 us; skew 500 ppm
I20240502 08:04:38.992966 6963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:04:38.995348 6963 webserver.cc:460] Webserver started at http://127.6.204.193:42997/ using document root <none> and password file <none>
I20240502 08:04:38.995754 6963 fs_manager.cc:359] Metadata directory not provided
I20240502 08:04:38.995903 6963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:04:38.996124 6963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:04:38.997232 6963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714636933625983-6963-0/minicluster-data/ts-0-root/instance:
uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3"
format_stamp: "Formatted at 2024-05-02 08:04:38 on dist-test-slave-32xs"
I20240502 08:04:39.001760 6963 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240502 08:04:39.005188 11748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:04:39.005937 6963 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20240502 08:04:39.006243 6963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714636933625983-6963-0/minicluster-data/ts-0-root
uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3"
format_stamp: "Formatted at 2024-05-02 08:04:38 on dist-test-slave-32xs"
I20240502 08:04:39.006522 6963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714636933625983-6963-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714636933625983-6963-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714636933625983-6963-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:04:39.024554 6963 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:04:39.025677 6963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:04:39.027205 6963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:04:39.029595 6963 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:04:39.029845 6963 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:04:39.030086 6963 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:04:39.030277 6963 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20240502 08:04:39.071316 6963 rpc_server.cc:307] RPC server started. Bound to: 127.6.204.193:38381
I20240502 08:04:39.071434 11810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.204.193:38381 every 8 connection(s)
I20240502 08:04:39.075986 6963 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:04:39.083982 11815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:04:39.084561 11816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:04:39.088138 11819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:04:39.089342 11811 heartbeater.cc:347] Connected to a master server at 127.6.204.254:38291
I20240502 08:04:39.089488 6963 server_base.cc:1034] running on GCE node
I20240502 08:04:39.089906 11811 heartbeater.cc:464] Registering TS with master...
I20240502 08:04:39.090611 6963 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:04:39.090838 6963 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:04:39.090724 11811 heartbeater.cc:511] Master 127.6.204.254:38291 requested a full tablet report, sending...
I20240502 08:04:39.091084 6963 hybrid_clock.cc:648] HybridClock initialized: now 1714637079091067 us; error 0 us; skew 500 ppm
I20240502 08:04:39.091758 6963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:04:39.093114 11684 ts_manager.cc:194] Registered new tserver with Master: c9eabd6e98644ebd8fcc4d9a9287cad3 (127.6.204.193:38381)
I20240502 08:04:39.094529 6963 webserver.cc:460] Webserver started at http://127.6.204.194:35657/ using document root <none> and password file <none>
I20240502 08:04:39.095001 6963 fs_manager.cc:359] Metadata directory not provided
I20240502 08:04:39.095187 6963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:04:39.095244 11684 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56272
I20240502 08:04:39.095453 6963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:04:39.097062 6963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714636933625983-6963-0/minicluster-data/ts-1-root/instance:
uuid: "1871d0a968764692b1a09a51fd0dab79"
format_stamp: "Formatted at 2024-05-02 08:04:39 on dist-test-slave-32xs"
I20240502 08:04:39.101622 6963 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20240502 08:04:39.104913 11823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:04:39.105684 6963 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 08:04:39.105955 6963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714636933625983-6963-0/minicluster-data/ts-1-root
uuid: "1871d0a968764692b1a09a51fd0dab79"
format_stamp: "Formatted at 2024-05-02 08:04:39 on dist-test-slave-32xs"
I20240502 08:04:39.106273 6963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714636933625983-6963-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714636933625983-6963-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714636933625983-6963-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:04:39.121138 6963 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:04:39.122370 6963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:04:39.123811 6963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:04:39.126067 6963 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:04:39.126312 6963 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:04:39.126539 6963 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:04:39.126690 6963 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:04:39.165902 6963 rpc_server.cc:307] RPC server started. Bound to: 127.6.204.194:39105
I20240502 08:04:39.165997 11885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.204.194:39105 every 8 connection(s)
I20240502 08:04:39.170948 6963 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:04:39.177549 11889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:04:39.180995 11890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:04:39.181640 6963 server_base.cc:1034] running on GCE node
W20240502 08:04:39.183044 11892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:04:39.184039 6963 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20240502 08:04:39.184201 11886 heartbeater.cc:347] Connected to a master server at 127.6.204.254:38291
W20240502 08:04:39.184304 6963 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:04:39.184653 6963 hybrid_clock.cc:648] HybridClock initialized: now 1714637079184634 us; error 0 us; skew 500 ppm
I20240502 08:04:39.184681 11886 heartbeater.cc:464] Registering TS with master...
I20240502 08:04:39.185292 6963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:04:39.185542 11886 heartbeater.cc:511] Master 127.6.204.254:38291 requested a full tablet report, sending...
I20240502 08:04:39.187664 11684 ts_manager.cc:194] Registered new tserver with Master: 1871d0a968764692b1a09a51fd0dab79 (127.6.204.194:39105)
I20240502 08:04:39.188046 6963 webserver.cc:460] Webserver started at http://127.6.204.195:45809/ using document root <none> and password file <none>
I20240502 08:04:39.188516 6963 fs_manager.cc:359] Metadata directory not provided
I20240502 08:04:39.188699 6963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:04:39.188946 6963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:04:39.189160 11684 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56282
I20240502 08:04:39.190227 6963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714636933625983-6963-0/minicluster-data/ts-2-root/instance:
uuid: "5f96a2946e9b4ae9b7bb144ee282b912"
format_stamp: "Formatted at 2024-05-02 08:04:39 on dist-test-slave-32xs"
I20240502 08:04:39.194778 6963 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.002s
I20240502 08:04:39.198271 11897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:04:39.199064 6963 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 08:04:39.199352 6963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714636933625983-6963-0/minicluster-data/ts-2-root
uuid: "5f96a2946e9b4ae9b7bb144ee282b912"
format_stamp: "Formatted at 2024-05-02 08:04:39 on dist-test-slave-32xs"
I20240502 08:04:39.199646 6963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714636933625983-6963-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714636933625983-6963-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714636933625983-6963-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:04:39.223994 6963 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:04:39.225147 6963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:04:39.226610 6963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:04:39.228844 6963 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:04:39.229040 6963 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:04:39.229267 6963 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:04:39.229429 6963 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:04:39.270020 6963 rpc_server.cc:307] RPC server started. Bound to: 127.6.204.195:40469
I20240502 08:04:39.270123 11959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.204.195:40469 every 8 connection(s)
I20240502 08:04:39.283334 11960 heartbeater.cc:347] Connected to a master server at 127.6.204.254:38291
I20240502 08:04:39.283699 11960 heartbeater.cc:464] Registering TS with master...
I20240502 08:04:39.284408 11960 heartbeater.cc:511] Master 127.6.204.254:38291 requested a full tablet report, sending...
I20240502 08:04:39.286238 11684 ts_manager.cc:194] Registered new tserver with Master: 5f96a2946e9b4ae9b7bb144ee282b912 (127.6.204.195:40469)
I20240502 08:04:39.286669 6963 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013473751s
I20240502 08:04:39.287981 11684 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56286
I20240502 08:04:40.098083 11811 heartbeater.cc:503] Master 127.6.204.254:38291 was elected leader, sending a full tablet report...
I20240502 08:04:40.191591 11886 heartbeater.cc:503] Master 127.6.204.254:38291 was elected leader, sending a full tablet report...
I20240502 08:04:40.290799 11960 heartbeater.cc:503] Master 127.6.204.254:38291 was elected leader, sending a full tablet report...
I20240502 08:04:40.320233 6963 test_util.cc:268] Using random seed: 236411088
I20240502 08:04:40.342823 11684 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56390:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:04:40.346570 11684 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:04:40.409471 11775 tablet_service.cc:1467] Processing CreateTablet for tablet d5f4dbf9e6f74bfc956ac9673786ccb5 (DEFAULT_TABLE table=test-workload [id=c8edd8b7337742c39f9e0e3f88156f20]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240502 08:04:40.409941 11776 tablet_service.cc:1467] Processing CreateTablet for tablet 20f890980e404e5ea6ec8995c97ed78c (DEFAULT_TABLE table=test-workload [id=c8edd8b7337742c39f9e0e3f88156f20]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240502 08:04:40.410923 11775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5f4dbf9e6f74bfc956ac9673786ccb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:40.411757 11776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20f890980e404e5ea6ec8995c97ed78c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:40.419706 11851 tablet_service.cc:1467] Processing CreateTablet for tablet 20f890980e404e5ea6ec8995c97ed78c (DEFAULT_TABLE table=test-workload [id=c8edd8b7337742c39f9e0e3f88156f20]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240502 08:04:40.421741 11851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20f890980e404e5ea6ec8995c97ed78c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:40.421873 11848 tablet_service.cc:1467] Processing CreateTablet for tablet 33dd558b4b60478c97aad83f82b9f1e5 (DEFAULT_TABLE table=test-workload [id=c8edd8b7337742c39f9e0e3f88156f20]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240502 08:04:40.423244 11848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33dd558b4b60478c97aad83f82b9f1e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:40.424484 11774 tablet_service.cc:1467] Processing CreateTablet for tablet bf974a7cca944db093d36241f22d663a (DEFAULT_TABLE table=test-workload [id=c8edd8b7337742c39f9e0e3f88156f20]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240502 08:04:40.425109 11850 tablet_service.cc:1467] Processing CreateTablet for tablet d5f4dbf9e6f74bfc956ac9673786ccb5 (DEFAULT_TABLE table=test-workload [id=c8edd8b7337742c39f9e0e3f88156f20]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240502 08:04:40.426430 11849 tablet_service.cc:1467] Processing CreateTablet for tablet bf974a7cca944db093d36241f22d663a (DEFAULT_TABLE table=test-workload [id=c8edd8b7337742c39f9e0e3f88156f20]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240502 08:04:40.427385 11773 tablet_service.cc:1467] Processing CreateTablet for tablet 33dd558b4b60478c97aad83f82b9f1e5 (DEFAULT_TABLE table=test-workload [id=c8edd8b7337742c39f9e0e3f88156f20]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240502 08:04:40.427861 11849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf974a7cca944db093d36241f22d663a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:40.425845 11774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf974a7cca944db093d36241f22d663a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:40.430666 11850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5f4dbf9e6f74bfc956ac9673786ccb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:40.430938 11773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33dd558b4b60478c97aad83f82b9f1e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:40.464324 11980 tablet_bootstrap.cc:492] T d5f4dbf9e6f74bfc956ac9673786ccb5 P c9eabd6e98644ebd8fcc4d9a9287cad3: Bootstrap starting.
I20240502 08:04:40.486302 11924 tablet_service.cc:1467] Processing CreateTablet for tablet d5f4dbf9e6f74bfc956ac9673786ccb5 (DEFAULT_TABLE table=test-workload [id=c8edd8b7337742c39f9e0e3f88156f20]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240502 08:04:40.487026 11925 tablet_service.cc:1467] Processing CreateTablet for tablet 20f890980e404e5ea6ec8995c97ed78c (DEFAULT_TABLE table=test-workload [id=c8edd8b7337742c39f9e0e3f88156f20]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240502 08:04:40.487841 11924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5f4dbf9e6f74bfc956ac9673786ccb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:40.488468 11925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20f890980e404e5ea6ec8995c97ed78c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:40.491297 11923 tablet_service.cc:1467] Processing CreateTablet for tablet bf974a7cca944db093d36241f22d663a (DEFAULT_TABLE table=test-workload [id=c8edd8b7337742c39f9e0e3f88156f20]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240502 08:04:40.492671 11923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf974a7cca944db093d36241f22d663a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:40.495350 11922 tablet_service.cc:1467] Processing CreateTablet for tablet 33dd558b4b60478c97aad83f82b9f1e5 (DEFAULT_TABLE table=test-workload [id=c8edd8b7337742c39f9e0e3f88156f20]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240502 08:04:40.496613 11922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33dd558b4b60478c97aad83f82b9f1e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:40.512012 11980 tablet_bootstrap.cc:654] T d5f4dbf9e6f74bfc956ac9673786ccb5 P c9eabd6e98644ebd8fcc4d9a9287cad3: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:40.512059 11981 tablet_bootstrap.cc:492] T 33dd558b4b60478c97aad83f82b9f1e5 P 1871d0a968764692b1a09a51fd0dab79: Bootstrap starting.
I20240502 08:04:40.514040 11982 tablet_bootstrap.cc:492] T 20f890980e404e5ea6ec8995c97ed78c P 5f96a2946e9b4ae9b7bb144ee282b912: Bootstrap starting.
I20240502 08:04:40.520103 11982 tablet_bootstrap.cc:654] T 20f890980e404e5ea6ec8995c97ed78c P 5f96a2946e9b4ae9b7bb144ee282b912: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:40.528646 11980 tablet_bootstrap.cc:492] T d5f4dbf9e6f74bfc956ac9673786ccb5 P c9eabd6e98644ebd8fcc4d9a9287cad3: No bootstrap required, opened a new log
I20240502 08:04:40.529057 11980 ts_tablet_manager.cc:1367] T d5f4dbf9e6f74bfc956ac9673786ccb5 P c9eabd6e98644ebd8fcc4d9a9287cad3: Time spent bootstrapping tablet: real 0.065s user 0.018s sys 0.003s
I20240502 08:04:40.525272 11981 tablet_bootstrap.cc:654] T 33dd558b4b60478c97aad83f82b9f1e5 P 1871d0a968764692b1a09a51fd0dab79: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:40.531497 11980 raft_consensus.cc:348] T d5f4dbf9e6f74bfc956ac9673786ccb5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
I20240502 08:04:40.532080 11980 raft_consensus.cc:374] T d5f4dbf9e6f74bfc956ac9673786ccb5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:40.532335 11980 raft_consensus.cc:729] T d5f4dbf9e6f74bfc956ac9673786ccb5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9eabd6e98644ebd8fcc4d9a9287cad3, State: Initialized, Role: FOLLOWER
I20240502 08:04:40.533064 11982 tablet_bootstrap.cc:492] T 20f890980e404e5ea6ec8995c97ed78c P 5f96a2946e9b4ae9b7bb144ee282b912: No bootstrap required, opened a new log
I20240502 08:04:40.532944 11980 consensus_queue.cc:260] T d5f4dbf9e6f74bfc956ac9673786ccb5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
I20240502 08:04:40.533756 11982 ts_tablet_manager.cc:1367] T 20f890980e404e5ea6ec8995c97ed78c P 5f96a2946e9b4ae9b7bb144ee282b912: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.008s
I20240502 08:04:40.535864 11981 tablet_bootstrap.cc:492] T 33dd558b4b60478c97aad83f82b9f1e5 P 1871d0a968764692b1a09a51fd0dab79: No bootstrap required, opened a new log
I20240502 08:04:40.535954 11980 ts_tablet_manager.cc:1398] T d5f4dbf9e6f74bfc956ac9673786ccb5 P c9eabd6e98644ebd8fcc4d9a9287cad3: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20240502 08:04:40.536442 11981 ts_tablet_manager.cc:1367] T 33dd558b4b60478c97aad83f82b9f1e5 P 1871d0a968764692b1a09a51fd0dab79: Time spent bootstrapping tablet: real 0.025s user 0.010s sys 0.003s
I20240502 08:04:40.536811 11980 tablet_bootstrap.cc:492] T 33dd558b4b60478c97aad83f82b9f1e5 P c9eabd6e98644ebd8fcc4d9a9287cad3: Bootstrap starting.
I20240502 08:04:40.538118 11982 raft_consensus.cc:348] T 20f890980e404e5ea6ec8995c97ed78c P 5f96a2946e9b4ae9b7bb144ee282b912 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
I20240502 08:04:40.538954 11982 raft_consensus.cc:374] T 20f890980e404e5ea6ec8995c97ed78c P 5f96a2946e9b4ae9b7bb144ee282b912 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:40.539306 11982 raft_consensus.cc:729] T 20f890980e404e5ea6ec8995c97ed78c P 5f96a2946e9b4ae9b7bb144ee282b912 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f96a2946e9b4ae9b7bb144ee282b912, State: Initialized, Role: FOLLOWER
I20240502 08:04:40.539322 11981 raft_consensus.cc:348] T 33dd558b4b60478c97aad83f82b9f1e5 P 1871d0a968764692b1a09a51fd0dab79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
I20240502 08:04:40.540278 11981 raft_consensus.cc:374] T 33dd558b4b60478c97aad83f82b9f1e5 P 1871d0a968764692b1a09a51fd0dab79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:40.540625 11981 raft_consensus.cc:729] T 33dd558b4b60478c97aad83f82b9f1e5 P 1871d0a968764692b1a09a51fd0dab79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1871d0a968764692b1a09a51fd0dab79, State: Initialized, Role: FOLLOWER
I20240502 08:04:40.540366 11982 consensus_queue.cc:260] T 20f890980e404e5ea6ec8995c97ed78c P 5f96a2946e9b4ae9b7bb144ee282b912 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
I20240502 08:04:40.541448 11981 consensus_queue.cc:260] T 33dd558b4b60478c97aad83f82b9f1e5 P 1871d0a968764692b1a09a51fd0dab79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
I20240502 08:04:40.543408 11980 tablet_bootstrap.cc:654] T 33dd558b4b60478c97aad83f82b9f1e5 P c9eabd6e98644ebd8fcc4d9a9287cad3: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:40.550427 11982 ts_tablet_manager.cc:1398] T 20f890980e404e5ea6ec8995c97ed78c P 5f96a2946e9b4ae9b7bb144ee282b912: Time spent starting tablet: real 0.016s user 0.011s sys 0.002s
I20240502 08:04:40.551319 11981 ts_tablet_manager.cc:1398] T 33dd558b4b60478c97aad83f82b9f1e5 P 1871d0a968764692b1a09a51fd0dab79: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20240502 08:04:40.553906 11982 tablet_bootstrap.cc:492] T 33dd558b4b60478c97aad83f82b9f1e5 P 5f96a2946e9b4ae9b7bb144ee282b912: Bootstrap starting.
I20240502 08:04:40.554505 11981 tablet_bootstrap.cc:492] T d5f4dbf9e6f74bfc956ac9673786ccb5 P 1871d0a968764692b1a09a51fd0dab79: Bootstrap starting.
I20240502 08:04:40.560801 11981 tablet_bootstrap.cc:654] T d5f4dbf9e6f74bfc956ac9673786ccb5 P 1871d0a968764692b1a09a51fd0dab79: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:40.562069 11982 tablet_bootstrap.cc:654] T 33dd558b4b60478c97aad83f82b9f1e5 P 5f96a2946e9b4ae9b7bb144ee282b912: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:40.562160 11980 tablet_bootstrap.cc:492] T 33dd558b4b60478c97aad83f82b9f1e5 P c9eabd6e98644ebd8fcc4d9a9287cad3: No bootstrap required, opened a new log
I20240502 08:04:40.562968 11980 ts_tablet_manager.cc:1367] T 33dd558b4b60478c97aad83f82b9f1e5 P c9eabd6e98644ebd8fcc4d9a9287cad3: Time spent bootstrapping tablet: real 0.026s user 0.011s sys 0.013s
I20240502 08:04:40.565654 11980 raft_consensus.cc:348] T 33dd558b4b60478c97aad83f82b9f1e5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
I20240502 08:04:40.566460 11980 raft_consensus.cc:374] T 33dd558b4b60478c97aad83f82b9f1e5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:40.566687 11981 tablet_bootstrap.cc:492] T d5f4dbf9e6f74bfc956ac9673786ccb5 P 1871d0a968764692b1a09a51fd0dab79: No bootstrap required, opened a new log
I20240502 08:04:40.566766 11980 raft_consensus.cc:729] T 33dd558b4b60478c97aad83f82b9f1e5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9eabd6e98644ebd8fcc4d9a9287cad3, State: Initialized, Role: FOLLOWER
I20240502 08:04:40.567297 11981 ts_tablet_manager.cc:1367] T d5f4dbf9e6f74bfc956ac9673786ccb5 P 1871d0a968764692b1a09a51fd0dab79: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.002s
I20240502 08:04:40.567669 11980 consensus_queue.cc:260] T 33dd558b4b60478c97aad83f82b9f1e5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
I20240502 08:04:40.569942 11980 ts_tablet_manager.cc:1398] T 33dd558b4b60478c97aad83f82b9f1e5 P c9eabd6e98644ebd8fcc4d9a9287cad3: Time spent starting tablet: real 0.007s user 0.003s sys 0.001s
I20240502 08:04:40.570907 11980 tablet_bootstrap.cc:492] T 20f890980e404e5ea6ec8995c97ed78c P c9eabd6e98644ebd8fcc4d9a9287cad3: Bootstrap starting.
I20240502 08:04:40.570791 11981 raft_consensus.cc:348] T d5f4dbf9e6f74bfc956ac9673786ccb5 P 1871d0a968764692b1a09a51fd0dab79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
I20240502 08:04:40.571931 11981 raft_consensus.cc:374] T d5f4dbf9e6f74bfc956ac9673786ccb5 P 1871d0a968764692b1a09a51fd0dab79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:40.572325 11981 raft_consensus.cc:729] T d5f4dbf9e6f74bfc956ac9673786ccb5 P 1871d0a968764692b1a09a51fd0dab79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1871d0a968764692b1a09a51fd0dab79, State: Initialized, Role: FOLLOWER
I20240502 08:04:40.573400 11981 consensus_queue.cc:260] T d5f4dbf9e6f74bfc956ac9673786ccb5 P 1871d0a968764692b1a09a51fd0dab79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
I20240502 08:04:40.576884 11980 tablet_bootstrap.cc:654] T 20f890980e404e5ea6ec8995c97ed78c P c9eabd6e98644ebd8fcc4d9a9287cad3: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:40.577204 11982 tablet_bootstrap.cc:492] T 33dd558b4b60478c97aad83f82b9f1e5 P 5f96a2946e9b4ae9b7bb144ee282b912: No bootstrap required, opened a new log
I20240502 08:04:40.577761 11982 ts_tablet_manager.cc:1367] T 33dd558b4b60478c97aad83f82b9f1e5 P 5f96a2946e9b4ae9b7bb144ee282b912: Time spent bootstrapping tablet: real 0.024s user 0.018s sys 0.000s
I20240502 08:04:40.577962 11981 ts_tablet_manager.cc:1398] T d5f4dbf9e6f74bfc956ac9673786ccb5 P 1871d0a968764692b1a09a51fd0dab79: Time spent starting tablet: real 0.010s user 0.006s sys 0.002s
I20240502 08:04:40.578958 11981 tablet_bootstrap.cc:492] T bf974a7cca944db093d36241f22d663a P 1871d0a968764692b1a09a51fd0dab79: Bootstrap starting.
I20240502 08:04:40.580796 11982 raft_consensus.cc:348] T 33dd558b4b60478c97aad83f82b9f1e5 P 5f96a2946e9b4ae9b7bb144ee282b912 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
I20240502 08:04:40.581738 11982 raft_consensus.cc:374] T 33dd558b4b60478c97aad83f82b9f1e5 P 5f96a2946e9b4ae9b7bb144ee282b912 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:40.582114 11982 raft_consensus.cc:729] T 33dd558b4b60478c97aad83f82b9f1e5 P 5f96a2946e9b4ae9b7bb144ee282b912 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f96a2946e9b4ae9b7bb144ee282b912, State: Initialized, Role: FOLLOWER
I20240502 08:04:40.583048 11982 consensus_queue.cc:260] T 33dd558b4b60478c97aad83f82b9f1e5 P 5f96a2946e9b4ae9b7bb144ee282b912 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
I20240502 08:04:40.584754 11981 tablet_bootstrap.cc:654] T bf974a7cca944db093d36241f22d663a P 1871d0a968764692b1a09a51fd0dab79: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:40.585642 11982 ts_tablet_manager.cc:1398] T 33dd558b4b60478c97aad83f82b9f1e5 P 5f96a2946e9b4ae9b7bb144ee282b912: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20240502 08:04:40.586691 11982 tablet_bootstrap.cc:492] T bf974a7cca944db093d36241f22d663a P 5f96a2946e9b4ae9b7bb144ee282b912: Bootstrap starting.
I20240502 08:04:40.593814 11982 tablet_bootstrap.cc:654] T bf974a7cca944db093d36241f22d663a P 5f96a2946e9b4ae9b7bb144ee282b912: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:40.596398 11981 tablet_bootstrap.cc:492] T bf974a7cca944db093d36241f22d663a P 1871d0a968764692b1a09a51fd0dab79: No bootstrap required, opened a new log
I20240502 08:04:40.601214 11981 ts_tablet_manager.cc:1367] T bf974a7cca944db093d36241f22d663a P 1871d0a968764692b1a09a51fd0dab79: Time spent bootstrapping tablet: real 0.022s user 0.010s sys 0.007s
I20240502 08:04:40.604146 11981 raft_consensus.cc:348] T bf974a7cca944db093d36241f22d663a P 1871d0a968764692b1a09a51fd0dab79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
I20240502 08:04:40.605146 11981 raft_consensus.cc:374] T bf974a7cca944db093d36241f22d663a P 1871d0a968764692b1a09a51fd0dab79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:40.605521 11981 raft_consensus.cc:729] T bf974a7cca944db093d36241f22d663a P 1871d0a968764692b1a09a51fd0dab79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1871d0a968764692b1a09a51fd0dab79, State: Initialized, Role: FOLLOWER
I20240502 08:04:40.606529 11981 consensus_queue.cc:260] T bf974a7cca944db093d36241f22d663a P 1871d0a968764692b1a09a51fd0dab79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
I20240502 08:04:40.609424 11981 ts_tablet_manager.cc:1398] T bf974a7cca944db093d36241f22d663a P 1871d0a968764692b1a09a51fd0dab79: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20240502 08:04:40.609897 11980 tablet_bootstrap.cc:492] T 20f890980e404e5ea6ec8995c97ed78c P c9eabd6e98644ebd8fcc4d9a9287cad3: No bootstrap required, opened a new log
I20240502 08:04:40.610075 11982 tablet_bootstrap.cc:492] T bf974a7cca944db093d36241f22d663a P 5f96a2946e9b4ae9b7bb144ee282b912: No bootstrap required, opened a new log
I20240502 08:04:40.610437 11980 ts_tablet_manager.cc:1367] T 20f890980e404e5ea6ec8995c97ed78c P c9eabd6e98644ebd8fcc4d9a9287cad3: Time spent bootstrapping tablet: real 0.040s user 0.019s sys 0.019s
I20240502 08:04:40.610455 11981 tablet_bootstrap.cc:492] T 20f890980e404e5ea6ec8995c97ed78c P 1871d0a968764692b1a09a51fd0dab79: Bootstrap starting.
I20240502 08:04:40.610702 11982 ts_tablet_manager.cc:1367] T bf974a7cca944db093d36241f22d663a P 5f96a2946e9b4ae9b7bb144ee282b912: Time spent bootstrapping tablet: real 0.024s user 0.015s sys 0.002s
I20240502 08:04:40.613391 11980 raft_consensus.cc:348] T 20f890980e404e5ea6ec8995c97ed78c P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
I20240502 08:04:40.614207 11980 raft_consensus.cc:374] T 20f890980e404e5ea6ec8995c97ed78c P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:40.613749 11982 raft_consensus.cc:348] T bf974a7cca944db093d36241f22d663a P 5f96a2946e9b4ae9b7bb144ee282b912 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
I20240502 08:04:40.614529 11980 raft_consensus.cc:729] T 20f890980e404e5ea6ec8995c97ed78c P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9eabd6e98644ebd8fcc4d9a9287cad3, State: Initialized, Role: FOLLOWER
I20240502 08:04:40.614874 11982 raft_consensus.cc:374] T bf974a7cca944db093d36241f22d663a P 5f96a2946e9b4ae9b7bb144ee282b912 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:40.615289 11982 raft_consensus.cc:729] T bf974a7cca944db093d36241f22d663a P 5f96a2946e9b4ae9b7bb144ee282b912 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f96a2946e9b4ae9b7bb144ee282b912, State: Initialized, Role: FOLLOWER
I20240502 08:04:40.615401 11980 consensus_queue.cc:260] T 20f890980e404e5ea6ec8995c97ed78c P c9eabd6e98644ebd8fcc4d9a9287cad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
I20240502 08:04:40.616297 11982 consensus_queue.cc:260] T bf974a7cca944db093d36241f22d663a P 5f96a2946e9b4ae9b7bb144ee282b912 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
I20240502 08:04:40.616935 11981 tablet_bootstrap.cc:654] T 20f890980e404e5ea6ec8995c97ed78c P 1871d0a968764692b1a09a51fd0dab79: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:40.617712 11980 ts_tablet_manager.cc:1398] T 20f890980e404e5ea6ec8995c97ed78c P c9eabd6e98644ebd8fcc4d9a9287cad3: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240502 08:04:40.618788 11980 tablet_bootstrap.cc:492] T bf974a7cca944db093d36241f22d663a P c9eabd6e98644ebd8fcc4d9a9287cad3: Bootstrap starting.
I20240502 08:04:40.621786 11982 ts_tablet_manager.cc:1398] T bf974a7cca944db093d36241f22d663a P 5f96a2946e9b4ae9b7bb144ee282b912: Time spent starting tablet: real 0.011s user 0.005s sys 0.002s
I20240502 08:04:40.623288 11982 tablet_bootstrap.cc:492] T d5f4dbf9e6f74bfc956ac9673786ccb5 P 5f96a2946e9b4ae9b7bb144ee282b912: Bootstrap starting.
I20240502 08:04:40.624719 11980 tablet_bootstrap.cc:654] T bf974a7cca944db093d36241f22d663a P c9eabd6e98644ebd8fcc4d9a9287cad3: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:40.625281 11986 raft_consensus.cc:482] T d5f4dbf9e6f74bfc956ac9673786ccb5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:04:40.625802 11986 raft_consensus.cc:504] T d5f4dbf9e6f74bfc956ac9673786ccb5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
I20240502 08:04:40.631371 11982 tablet_bootstrap.cc:654] T d5f4dbf9e6f74bfc956ac9673786ccb5 P 5f96a2946e9b4ae9b7bb144ee282b912: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:40.636703 11980 tablet_bootstrap.cc:492] T bf974a7cca944db093d36241f22d663a P c9eabd6e98644ebd8fcc4d9a9287cad3: No bootstrap required, opened a new log
I20240502 08:04:40.637205 11980 ts_tablet_manager.cc:1367] T bf974a7cca944db093d36241f22d663a P c9eabd6e98644ebd8fcc4d9a9287cad3: Time spent bootstrapping tablet: real 0.019s user 0.011s sys 0.005s
I20240502 08:04:40.640231 11980 raft_consensus.cc:348] T bf974a7cca944db093d36241f22d663a P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
I20240502 08:04:40.640954 11980 raft_consensus.cc:374] T bf974a7cca944db093d36241f22d663a P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:40.641238 11980 raft_consensus.cc:729] T bf974a7cca944db093d36241f22d663a P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9eabd6e98644ebd8fcc4d9a9287cad3, State: Initialized, Role: FOLLOWER
I20240502 08:04:40.642021 11980 consensus_queue.cc:260] T bf974a7cca944db093d36241f22d663a P c9eabd6e98644ebd8fcc4d9a9287cad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
I20240502 08:04:40.646811 11981 tablet_bootstrap.cc:492] T 20f890980e404e5ea6ec8995c97ed78c P 1871d0a968764692b1a09a51fd0dab79: No bootstrap required, opened a new log
I20240502 08:04:40.647486 11981 ts_tablet_manager.cc:1367] T 20f890980e404e5ea6ec8995c97ed78c P 1871d0a968764692b1a09a51fd0dab79: Time spent bootstrapping tablet: real 0.037s user 0.016s sys 0.019s
I20240502 08:04:40.649796 11980 ts_tablet_manager.cc:1398] T bf974a7cca944db093d36241f22d663a P c9eabd6e98644ebd8fcc4d9a9287cad3: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20240502 08:04:40.651165 11981 raft_consensus.cc:348] T 20f890980e404e5ea6ec8995c97ed78c P 1871d0a968764692b1a09a51fd0dab79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
I20240502 08:04:40.652230 11981 raft_consensus.cc:374] T 20f890980e404e5ea6ec8995c97ed78c P 1871d0a968764692b1a09a51fd0dab79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:40.652643 11981 raft_consensus.cc:729] T 20f890980e404e5ea6ec8995c97ed78c P 1871d0a968764692b1a09a51fd0dab79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1871d0a968764692b1a09a51fd0dab79, State: Initialized, Role: FOLLOWER
I20240502 08:04:40.653594 11981 consensus_queue.cc:260] T 20f890980e404e5ea6ec8995c97ed78c P 1871d0a968764692b1a09a51fd0dab79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
I20240502 08:04:40.659135 11981 ts_tablet_manager.cc:1398] T 20f890980e404e5ea6ec8995c97ed78c P 1871d0a968764692b1a09a51fd0dab79: Time spent starting tablet: real 0.011s user 0.006s sys 0.003s
I20240502 08:04:40.665937 11986 leader_election.cc:288] T d5f4dbf9e6f74bfc956ac9673786ccb5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1871d0a968764692b1a09a51fd0dab79 (127.6.204.194:39105), 5f96a2946e9b4ae9b7bb144ee282b912 (127.6.204.195:40469)
I20240502 08:04:40.675746 11982 tablet_bootstrap.cc:492] T d5f4dbf9e6f74bfc956ac9673786ccb5 P 5f96a2946e9b4ae9b7bb144ee282b912: No bootstrap required, opened a new log
I20240502 08:04:40.676234 11982 ts_tablet_manager.cc:1367] T d5f4dbf9e6f74bfc956ac9673786ccb5 P 5f96a2946e9b4ae9b7bb144ee282b912: Time spent bootstrapping tablet: real 0.053s user 0.031s sys 0.018s
I20240502 08:04:40.678354 11982 raft_consensus.cc:348] T d5f4dbf9e6f74bfc956ac9673786ccb5 P 5f96a2946e9b4ae9b7bb144ee282b912 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
I20240502 08:04:40.679092 11982 raft_consensus.cc:374] T d5f4dbf9e6f74bfc956ac9673786ccb5 P 5f96a2946e9b4ae9b7bb144ee282b912 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:40.679383 11982 raft_consensus.cc:729] T d5f4dbf9e6f74bfc956ac9673786ccb5 P 5f96a2946e9b4ae9b7bb144ee282b912 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f96a2946e9b4ae9b7bb144ee282b912, State: Initialized, Role: FOLLOWER
I20240502 08:04:40.680126 11982 consensus_queue.cc:260] T d5f4dbf9e6f74bfc956ac9673786ccb5 P 5f96a2946e9b4ae9b7bb144ee282b912 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
I20240502 08:04:40.681439 11935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5f4dbf9e6f74bfc956ac9673786ccb5" candidate_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" is_pre_election: true
I20240502 08:04:40.682309 11935 raft_consensus.cc:2454] T d5f4dbf9e6f74bfc956ac9673786ccb5 P 5f96a2946e9b4ae9b7bb144ee282b912 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9eabd6e98644ebd8fcc4d9a9287cad3 in term 0.
I20240502 08:04:40.682754 11861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5f4dbf9e6f74bfc956ac9673786ccb5" candidate_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1871d0a968764692b1a09a51fd0dab79" is_pre_election: true
I20240502 08:04:40.683385 11861 raft_consensus.cc:2454] T d5f4dbf9e6f74bfc956ac9673786ccb5 P 1871d0a968764692b1a09a51fd0dab79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9eabd6e98644ebd8fcc4d9a9287cad3 in term 0.
I20240502 08:04:40.683738 11749 leader_election.cc:302] T d5f4dbf9e6f74bfc956ac9673786ccb5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f96a2946e9b4ae9b7bb144ee282b912, c9eabd6e98644ebd8fcc4d9a9287cad3; no voters:
I20240502 08:04:40.684105 11982 ts_tablet_manager.cc:1398] T d5f4dbf9e6f74bfc956ac9673786ccb5 P 5f96a2946e9b4ae9b7bb144ee282b912: Time spent starting tablet: real 0.008s user 0.006s sys 0.002s
I20240502 08:04:40.684522 11986 raft_consensus.cc:2781] T d5f4dbf9e6f74bfc956ac9673786ccb5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:04:40.684851 11986 raft_consensus.cc:482] T d5f4dbf9e6f74bfc956ac9673786ccb5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:04:40.685066 11986 raft_consensus.cc:3037] T d5f4dbf9e6f74bfc956ac9673786ccb5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:40.689401 11998 raft_consensus.cc:482] T bf974a7cca944db093d36241f22d663a P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:04:40.689872 11998 raft_consensus.cc:504] T bf974a7cca944db093d36241f22d663a P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
I20240502 08:04:40.690848 11986 raft_consensus.cc:504] T d5f4dbf9e6f74bfc956ac9673786ccb5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
I20240502 08:04:40.691650 11998 leader_election.cc:288] T bf974a7cca944db093d36241f22d663a P c9eabd6e98644ebd8fcc4d9a9287cad3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1871d0a968764692b1a09a51fd0dab79 (127.6.204.194:39105), 5f96a2946e9b4ae9b7bb144ee282b912 (127.6.204.195:40469)
I20240502 08:04:40.692400 11861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf974a7cca944db093d36241f22d663a" candidate_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1871d0a968764692b1a09a51fd0dab79" is_pre_election: true
I20240502 08:04:40.692577 11935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf974a7cca944db093d36241f22d663a" candidate_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" is_pre_election: true
I20240502 08:04:40.693065 11861 raft_consensus.cc:2454] T bf974a7cca944db093d36241f22d663a P 1871d0a968764692b1a09a51fd0dab79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9eabd6e98644ebd8fcc4d9a9287cad3 in term 0.
I20240502 08:04:40.693121 11935 raft_consensus.cc:2454] T bf974a7cca944db093d36241f22d663a P 5f96a2946e9b4ae9b7bb144ee282b912 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9eabd6e98644ebd8fcc4d9a9287cad3 in term 0.
I20240502 08:04:40.694314 11749 leader_election.cc:302] T bf974a7cca944db093d36241f22d663a P c9eabd6e98644ebd8fcc4d9a9287cad3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f96a2946e9b4ae9b7bb144ee282b912, c9eabd6e98644ebd8fcc4d9a9287cad3; no voters:
I20240502 08:04:40.694972 11986 leader_election.cc:288] T d5f4dbf9e6f74bfc956ac9673786ccb5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [CANDIDATE]: Term 1 election: Requested vote from peers 1871d0a968764692b1a09a51fd0dab79 (127.6.204.194:39105), 5f96a2946e9b4ae9b7bb144ee282b912 (127.6.204.195:40469)
I20240502 08:04:40.694957 11998 raft_consensus.cc:2781] T bf974a7cca944db093d36241f22d663a P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:04:40.695827 11998 raft_consensus.cc:482] T bf974a7cca944db093d36241f22d663a P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:04:40.695811 11861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5f4dbf9e6f74bfc956ac9673786ccb5" candidate_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1871d0a968764692b1a09a51fd0dab79"
I20240502 08:04:40.695968 11935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5f4dbf9e6f74bfc956ac9673786ccb5" candidate_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f96a2946e9b4ae9b7bb144ee282b912"
I20240502 08:04:40.696439 11998 raft_consensus.cc:3037] T bf974a7cca944db093d36241f22d663a P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:40.696576 11861 raft_consensus.cc:3037] T d5f4dbf9e6f74bfc956ac9673786ccb5 P 1871d0a968764692b1a09a51fd0dab79 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:40.696653 11935 raft_consensus.cc:3037] T d5f4dbf9e6f74bfc956ac9673786ccb5 P 5f96a2946e9b4ae9b7bb144ee282b912 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:40.702124 11935 raft_consensus.cc:2454] T d5f4dbf9e6f74bfc956ac9673786ccb5 P 5f96a2946e9b4ae9b7bb144ee282b912 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9eabd6e98644ebd8fcc4d9a9287cad3 in term 1.
I20240502 08:04:40.703035 11749 leader_election.cc:302] T d5f4dbf9e6f74bfc956ac9673786ccb5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f96a2946e9b4ae9b7bb144ee282b912, c9eabd6e98644ebd8fcc4d9a9287cad3; no voters:
I20240502 08:04:40.703698 11986 raft_consensus.cc:2781] T d5f4dbf9e6f74bfc956ac9673786ccb5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:04:40.703892 11861 raft_consensus.cc:2454] T d5f4dbf9e6f74bfc956ac9673786ccb5 P 1871d0a968764692b1a09a51fd0dab79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9eabd6e98644ebd8fcc4d9a9287cad3 in term 1.
I20240502 08:04:40.705009 11986 raft_consensus.cc:686] T d5f4dbf9e6f74bfc956ac9673786ccb5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 1 LEADER]: Becoming Leader. State: Replica: c9eabd6e98644ebd8fcc4d9a9287cad3, State: Running, Role: LEADER
I20240502 08:04:40.705245 11998 raft_consensus.cc:504] T bf974a7cca944db093d36241f22d663a P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
I20240502 08:04:40.705947 11986 consensus_queue.cc:237] T d5f4dbf9e6f74bfc956ac9673786ccb5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
I20240502 08:04:40.708032 11861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf974a7cca944db093d36241f22d663a" candidate_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1871d0a968764692b1a09a51fd0dab79"
I20240502 08:04:40.708642 11861 raft_consensus.cc:3037] T bf974a7cca944db093d36241f22d663a P 1871d0a968764692b1a09a51fd0dab79 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:40.709110 11935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf974a7cca944db093d36241f22d663a" candidate_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f96a2946e9b4ae9b7bb144ee282b912"
I20240502 08:04:40.709872 11935 raft_consensus.cc:3037] T bf974a7cca944db093d36241f22d663a P 5f96a2946e9b4ae9b7bb144ee282b912 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:40.710534 11998 leader_election.cc:288] T bf974a7cca944db093d36241f22d663a P c9eabd6e98644ebd8fcc4d9a9287cad3 [CANDIDATE]: Term 1 election: Requested vote from peers 1871d0a968764692b1a09a51fd0dab79 (127.6.204.194:39105), 5f96a2946e9b4ae9b7bb144ee282b912 (127.6.204.195:40469)
I20240502 08:04:40.714118 11861 raft_consensus.cc:2454] T bf974a7cca944db093d36241f22d663a P 1871d0a968764692b1a09a51fd0dab79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9eabd6e98644ebd8fcc4d9a9287cad3 in term 1.
I20240502 08:04:40.715106 12002 raft_consensus.cc:482] T 33dd558b4b60478c97aad83f82b9f1e5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:04:40.715866 11752 leader_election.cc:302] T bf974a7cca944db093d36241f22d663a P c9eabd6e98644ebd8fcc4d9a9287cad3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1871d0a968764692b1a09a51fd0dab79, c9eabd6e98644ebd8fcc4d9a9287cad3; no voters:
I20240502 08:04:40.715765 12002 raft_consensus.cc:504] T 33dd558b4b60478c97aad83f82b9f1e5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
I20240502 08:04:40.716933 11986 raft_consensus.cc:2781] T bf974a7cca944db093d36241f22d663a P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:04:40.717469 11986 raft_consensus.cc:686] T bf974a7cca944db093d36241f22d663a P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 1 LEADER]: Becoming Leader. State: Replica: c9eabd6e98644ebd8fcc4d9a9287cad3, State: Running, Role: LEADER
I20240502 08:04:40.718338 11986 consensus_queue.cc:237] T bf974a7cca944db093d36241f22d663a P c9eabd6e98644ebd8fcc4d9a9287cad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
I20240502 08:04:40.719010 11861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33dd558b4b60478c97aad83f82b9f1e5" candidate_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1871d0a968764692b1a09a51fd0dab79" is_pre_election: true
I20240502 08:04:40.719717 11861 raft_consensus.cc:2454] T 33dd558b4b60478c97aad83f82b9f1e5 P 1871d0a968764692b1a09a51fd0dab79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9eabd6e98644ebd8fcc4d9a9287cad3 in term 0.
I20240502 08:04:40.720036 11934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33dd558b4b60478c97aad83f82b9f1e5" candidate_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" is_pre_election: true
I20240502 08:04:40.720449 12002 leader_election.cc:288] T 33dd558b4b60478c97aad83f82b9f1e5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1871d0a968764692b1a09a51fd0dab79 (127.6.204.194:39105), 5f96a2946e9b4ae9b7bb144ee282b912 (127.6.204.195:40469)
I20240502 08:04:40.720693 11934 raft_consensus.cc:2454] T 33dd558b4b60478c97aad83f82b9f1e5 P 5f96a2946e9b4ae9b7bb144ee282b912 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9eabd6e98644ebd8fcc4d9a9287cad3 in term 0.
I20240502 08:04:40.721824 11749 leader_election.cc:302] T 33dd558b4b60478c97aad83f82b9f1e5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f96a2946e9b4ae9b7bb144ee282b912, c9eabd6e98644ebd8fcc4d9a9287cad3; no voters:
I20240502 08:04:40.722671 12002 raft_consensus.cc:2781] T 33dd558b4b60478c97aad83f82b9f1e5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:04:40.722996 12002 raft_consensus.cc:482] T 33dd558b4b60478c97aad83f82b9f1e5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:04:40.723320 12002 raft_consensus.cc:3037] T 33dd558b4b60478c97aad83f82b9f1e5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:40.723526 11935 raft_consensus.cc:2454] T bf974a7cca944db093d36241f22d663a P 5f96a2946e9b4ae9b7bb144ee282b912 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9eabd6e98644ebd8fcc4d9a9287cad3 in term 1.
I20240502 08:04:40.724056 11684 catalog_manager.cc:5484] T d5f4dbf9e6f74bfc956ac9673786ccb5 P c9eabd6e98644ebd8fcc4d9a9287cad3 reported cstate change: term changed from 0 to 1, leader changed from <none> to c9eabd6e98644ebd8fcc4d9a9287cad3 (127.6.204.193). New cstate: current_term: 1 leader_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } health_report { overall_health: UNKNOWN } } }
I20240502 08:04:40.731268 12002 raft_consensus.cc:504] T 33dd558b4b60478c97aad83f82b9f1e5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
I20240502 08:04:40.734040 11861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33dd558b4b60478c97aad83f82b9f1e5" candidate_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1871d0a968764692b1a09a51fd0dab79"
I20240502 08:04:40.734090 11935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33dd558b4b60478c97aad83f82b9f1e5" candidate_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f96a2946e9b4ae9b7bb144ee282b912"
I20240502 08:04:40.734812 11861 raft_consensus.cc:3037] T 33dd558b4b60478c97aad83f82b9f1e5 P 1871d0a968764692b1a09a51fd0dab79 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:40.734889 11935 raft_consensus.cc:3037] T 33dd558b4b60478c97aad83f82b9f1e5 P 5f96a2946e9b4ae9b7bb144ee282b912 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:40.739015 11684 catalog_manager.cc:5484] T bf974a7cca944db093d36241f22d663a P c9eabd6e98644ebd8fcc4d9a9287cad3 reported cstate change: term changed from 0 to 1, leader changed from <none> to c9eabd6e98644ebd8fcc4d9a9287cad3 (127.6.204.193). New cstate: current_term: 1 leader_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } health_report { overall_health: UNKNOWN } } }
I20240502 08:04:40.740718 11935 raft_consensus.cc:2454] T 33dd558b4b60478c97aad83f82b9f1e5 P 5f96a2946e9b4ae9b7bb144ee282b912 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9eabd6e98644ebd8fcc4d9a9287cad3 in term 1.
I20240502 08:04:40.741369 11861 raft_consensus.cc:2454] T 33dd558b4b60478c97aad83f82b9f1e5 P 1871d0a968764692b1a09a51fd0dab79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9eabd6e98644ebd8fcc4d9a9287cad3 in term 1.
I20240502 08:04:40.741681 11749 leader_election.cc:302] T 33dd558b4b60478c97aad83f82b9f1e5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f96a2946e9b4ae9b7bb144ee282b912, c9eabd6e98644ebd8fcc4d9a9287cad3; no voters:
I20240502 08:04:40.742231 12002 leader_election.cc:288] T 33dd558b4b60478c97aad83f82b9f1e5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [CANDIDATE]: Term 1 election: Requested vote from peers 1871d0a968764692b1a09a51fd0dab79 (127.6.204.194:39105), 5f96a2946e9b4ae9b7bb144ee282b912 (127.6.204.195:40469)
I20240502 08:04:40.742825 11986 raft_consensus.cc:2781] T 33dd558b4b60478c97aad83f82b9f1e5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:04:40.743336 11986 raft_consensus.cc:686] T 33dd558b4b60478c97aad83f82b9f1e5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 1 LEADER]: Becoming Leader. State: Replica: c9eabd6e98644ebd8fcc4d9a9287cad3, State: Running, Role: LEADER
I20240502 08:04:40.744216 11986 consensus_queue.cc:237] T 33dd558b4b60478c97aad83f82b9f1e5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
I20240502 08:04:40.752538 11684 catalog_manager.cc:5484] T 33dd558b4b60478c97aad83f82b9f1e5 P c9eabd6e98644ebd8fcc4d9a9287cad3 reported cstate change: term changed from 0 to 1, leader changed from <none> to c9eabd6e98644ebd8fcc4d9a9287cad3 (127.6.204.193). New cstate: current_term: 1 leader_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } health_report { overall_health: UNKNOWN } } }
I20240502 08:04:40.808517 11986 raft_consensus.cc:482] T 20f890980e404e5ea6ec8995c97ed78c P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:04:40.809000 11986 raft_consensus.cc:504] T 20f890980e404e5ea6ec8995c97ed78c P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
I20240502 08:04:40.810678 11986 leader_election.cc:288] T 20f890980e404e5ea6ec8995c97ed78c P c9eabd6e98644ebd8fcc4d9a9287cad3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1871d0a968764692b1a09a51fd0dab79 (127.6.204.194:39105), 5f96a2946e9b4ae9b7bb144ee282b912 (127.6.204.195:40469)
I20240502 08:04:40.811535 11861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20f890980e404e5ea6ec8995c97ed78c" candidate_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1871d0a968764692b1a09a51fd0dab79" is_pre_election: true
I20240502 08:04:40.811726 11935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20f890980e404e5ea6ec8995c97ed78c" candidate_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" is_pre_election: true
I20240502 08:04:40.812095 11861 raft_consensus.cc:2454] T 20f890980e404e5ea6ec8995c97ed78c P 1871d0a968764692b1a09a51fd0dab79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9eabd6e98644ebd8fcc4d9a9287cad3 in term 0.
I20240502 08:04:40.812301 11935 raft_consensus.cc:2454] T 20f890980e404e5ea6ec8995c97ed78c P 5f96a2946e9b4ae9b7bb144ee282b912 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9eabd6e98644ebd8fcc4d9a9287cad3 in term 0.
I20240502 08:04:40.812997 11752 leader_election.cc:302] T 20f890980e404e5ea6ec8995c97ed78c P c9eabd6e98644ebd8fcc4d9a9287cad3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1871d0a968764692b1a09a51fd0dab79, c9eabd6e98644ebd8fcc4d9a9287cad3; no voters:
I20240502 08:04:40.813635 11986 raft_consensus.cc:2781] T 20f890980e404e5ea6ec8995c97ed78c P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:04:40.813918 11986 raft_consensus.cc:482] T 20f890980e404e5ea6ec8995c97ed78c P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:04:40.814199 11986 raft_consensus.cc:3037] T 20f890980e404e5ea6ec8995c97ed78c P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:40.818943 11986 raft_consensus.cc:504] T 20f890980e404e5ea6ec8995c97ed78c P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
I20240502 08:04:40.820408 11986 leader_election.cc:288] T 20f890980e404e5ea6ec8995c97ed78c P c9eabd6e98644ebd8fcc4d9a9287cad3 [CANDIDATE]: Term 1 election: Requested vote from peers 1871d0a968764692b1a09a51fd0dab79 (127.6.204.194:39105), 5f96a2946e9b4ae9b7bb144ee282b912 (127.6.204.195:40469)
I20240502 08:04:40.821260 11861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20f890980e404e5ea6ec8995c97ed78c" candidate_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1871d0a968764692b1a09a51fd0dab79"
I20240502 08:04:40.821367 11935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20f890980e404e5ea6ec8995c97ed78c" candidate_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f96a2946e9b4ae9b7bb144ee282b912"
I20240502 08:04:40.821808 11861 raft_consensus.cc:3037] T 20f890980e404e5ea6ec8995c97ed78c P 1871d0a968764692b1a09a51fd0dab79 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:40.821902 11935 raft_consensus.cc:3037] T 20f890980e404e5ea6ec8995c97ed78c P 5f96a2946e9b4ae9b7bb144ee282b912 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:40.826473 11935 raft_consensus.cc:2454] T 20f890980e404e5ea6ec8995c97ed78c P 5f96a2946e9b4ae9b7bb144ee282b912 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9eabd6e98644ebd8fcc4d9a9287cad3 in term 1.
I20240502 08:04:40.826472 11861 raft_consensus.cc:2454] T 20f890980e404e5ea6ec8995c97ed78c P 1871d0a968764692b1a09a51fd0dab79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9eabd6e98644ebd8fcc4d9a9287cad3 in term 1.
I20240502 08:04:40.827628 11749 leader_election.cc:302] T 20f890980e404e5ea6ec8995c97ed78c P c9eabd6e98644ebd8fcc4d9a9287cad3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f96a2946e9b4ae9b7bb144ee282b912, c9eabd6e98644ebd8fcc4d9a9287cad3; no voters:
I20240502 08:04:40.828328 11986 raft_consensus.cc:2781] T 20f890980e404e5ea6ec8995c97ed78c P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:04:40.828692 11986 raft_consensus.cc:686] T 20f890980e404e5ea6ec8995c97ed78c P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 1 LEADER]: Becoming Leader. State: Replica: c9eabd6e98644ebd8fcc4d9a9287cad3, State: Running, Role: LEADER
I20240502 08:04:40.829370 11986 consensus_queue.cc:237] T 20f890980e404e5ea6ec8995c97ed78c P c9eabd6e98644ebd8fcc4d9a9287cad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
I20240502 08:04:40.836375 11684 catalog_manager.cc:5484] T 20f890980e404e5ea6ec8995c97ed78c P c9eabd6e98644ebd8fcc4d9a9287cad3 reported cstate change: term changed from 0 to 1, leader changed from <none> to c9eabd6e98644ebd8fcc4d9a9287cad3 (127.6.204.193). New cstate: current_term: 1 leader_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } health_report { overall_health: UNKNOWN } } }
I20240502 08:04:40.872361 6963 tablet_server.cc:178] TabletServer@127.6.204.193:0 shutting down...
I20240502 08:04:40.890074 6963 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 08:04:40.890915 6963 tablet_replica.cc:330] T bf974a7cca944db093d36241f22d663a P c9eabd6e98644ebd8fcc4d9a9287cad3: stopping tablet replica
I20240502 08:04:40.891670 6963 raft_consensus.cc:2229] T bf974a7cca944db093d36241f22d663a P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:04:40.892433 6963 pending_rounds.cc:62] T bf974a7cca944db093d36241f22d663a P c9eabd6e98644ebd8fcc4d9a9287cad3: Trying to abort 1 pending ops.
I20240502 08:04:40.892690 6963 pending_rounds.cc:68] T bf974a7cca944db093d36241f22d663a P c9eabd6e98644ebd8fcc4d9a9287cad3: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023153482632065024 op_type: NO_OP noop_request { }
I20240502 08:04:40.893111 6963 raft_consensus.cc:2866] T bf974a7cca944db093d36241f22d663a P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240502 08:04:40.893435 6963 raft_consensus.cc:2258] T bf974a7cca944db093d36241f22d663a P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:04:40.895817 6963 tablet_replica.cc:330] T 33dd558b4b60478c97aad83f82b9f1e5 P c9eabd6e98644ebd8fcc4d9a9287cad3: stopping tablet replica
I20240502 08:04:40.896306 6963 raft_consensus.cc:2229] T 33dd558b4b60478c97aad83f82b9f1e5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:04:40.896795 6963 pending_rounds.cc:62] T 33dd558b4b60478c97aad83f82b9f1e5 P c9eabd6e98644ebd8fcc4d9a9287cad3: Trying to abort 1 pending ops.
I20240502 08:04:40.896970 6963 pending_rounds.cc:68] T 33dd558b4b60478c97aad83f82b9f1e5 P c9eabd6e98644ebd8fcc4d9a9287cad3: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023153482736619520 op_type: NO_OP noop_request { }
I20240502 08:04:40.897228 6963 raft_consensus.cc:2866] T 33dd558b4b60478c97aad83f82b9f1e5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240502 08:04:40.897444 6963 raft_consensus.cc:2258] T 33dd558b4b60478c97aad83f82b9f1e5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:04:40.899593 6963 tablet_replica.cc:330] T 20f890980e404e5ea6ec8995c97ed78c P c9eabd6e98644ebd8fcc4d9a9287cad3: stopping tablet replica
I20240502 08:04:40.900082 6963 raft_consensus.cc:2229] T 20f890980e404e5ea6ec8995c97ed78c P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:04:40.900571 6963 pending_rounds.cc:62] T 20f890980e404e5ea6ec8995c97ed78c P c9eabd6e98644ebd8fcc4d9a9287cad3: Trying to abort 1 pending ops.
I20240502 08:04:40.900727 6963 pending_rounds.cc:68] T 20f890980e404e5ea6ec8995c97ed78c P c9eabd6e98644ebd8fcc4d9a9287cad3: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023153483083653120 op_type: NO_OP noop_request { }
I20240502 08:04:40.901005 6963 raft_consensus.cc:2866] T 20f890980e404e5ea6ec8995c97ed78c P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240502 08:04:40.901279 6963 raft_consensus.cc:2258] T 20f890980e404e5ea6ec8995c97ed78c P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:04:40.903406 6963 tablet_replica.cc:330] T d5f4dbf9e6f74bfc956ac9673786ccb5 P c9eabd6e98644ebd8fcc4d9a9287cad3: stopping tablet replica
I20240502 08:04:40.903846 6963 raft_consensus.cc:2229] T d5f4dbf9e6f74bfc956ac9673786ccb5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:04:40.904366 6963 pending_rounds.cc:62] T d5f4dbf9e6f74bfc956ac9673786ccb5 P c9eabd6e98644ebd8fcc4d9a9287cad3: Trying to abort 1 pending ops.
I20240502 08:04:40.904515 6963 pending_rounds.cc:68] T d5f4dbf9e6f74bfc956ac9673786ccb5 P c9eabd6e98644ebd8fcc4d9a9287cad3: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023153482581434368 op_type: NO_OP noop_request { }
I20240502 08:04:40.904776 6963 raft_consensus.cc:2866] T d5f4dbf9e6f74bfc956ac9673786ccb5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240502 08:04:40.905010 6963 raft_consensus.cc:2258] T d5f4dbf9e6f74bfc956ac9673786ccb5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:04:40.918576 6963 tablet_server.cc:194] TabletServer@127.6.204.193:0 shutdown complete.
I20240502 08:04:40.929147 6963 tablet_server.cc:178] TabletServer@127.6.204.194:0 shutting down...
I20240502 08:04:40.947708 6963 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 08:04:40.948518 6963 tablet_replica.cc:330] T bf974a7cca944db093d36241f22d663a P 1871d0a968764692b1a09a51fd0dab79: stopping tablet replica
I20240502 08:04:40.949065 6963 raft_consensus.cc:2229] T bf974a7cca944db093d36241f22d663a P 1871d0a968764692b1a09a51fd0dab79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:04:40.949452 6963 raft_consensus.cc:2258] T bf974a7cca944db093d36241f22d663a P 1871d0a968764692b1a09a51fd0dab79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:04:40.951290 6963 tablet_replica.cc:330] T 20f890980e404e5ea6ec8995c97ed78c P 1871d0a968764692b1a09a51fd0dab79: stopping tablet replica
I20240502 08:04:40.951779 6963 raft_consensus.cc:2229] T 20f890980e404e5ea6ec8995c97ed78c P 1871d0a968764692b1a09a51fd0dab79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:04:40.952128 6963 raft_consensus.cc:2258] T 20f890980e404e5ea6ec8995c97ed78c P 1871d0a968764692b1a09a51fd0dab79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:04:40.953882 6963 tablet_replica.cc:330] T d5f4dbf9e6f74bfc956ac9673786ccb5 P 1871d0a968764692b1a09a51fd0dab79: stopping tablet replica
I20240502 08:04:40.954408 6963 raft_consensus.cc:2229] T d5f4dbf9e6f74bfc956ac9673786ccb5 P 1871d0a968764692b1a09a51fd0dab79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:04:40.954754 6963 raft_consensus.cc:2258] T d5f4dbf9e6f74bfc956ac9673786ccb5 P 1871d0a968764692b1a09a51fd0dab79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:04:40.956466 6963 tablet_replica.cc:330] T 33dd558b4b60478c97aad83f82b9f1e5 P 1871d0a968764692b1a09a51fd0dab79: stopping tablet replica
I20240502 08:04:40.956894 6963 raft_consensus.cc:2229] T 33dd558b4b60478c97aad83f82b9f1e5 P 1871d0a968764692b1a09a51fd0dab79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:04:40.957226 6963 raft_consensus.cc:2258] T 33dd558b4b60478c97aad83f82b9f1e5 P 1871d0a968764692b1a09a51fd0dab79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:04:40.977907 6963 tablet_server.cc:194] TabletServer@127.6.204.194:0 shutdown complete.
I20240502 08:04:40.989151 6963 tablet_server.cc:178] TabletServer@127.6.204.195:0 shutting down...
I20240502 08:04:41.005610 6963 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 08:04:41.006408 6963 tablet_replica.cc:330] T bf974a7cca944db093d36241f22d663a P 5f96a2946e9b4ae9b7bb144ee282b912: stopping tablet replica
I20240502 08:04:41.006989 6963 raft_consensus.cc:2229] T bf974a7cca944db093d36241f22d663a P 5f96a2946e9b4ae9b7bb144ee282b912 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:04:41.007380 6963 raft_consensus.cc:2258] T bf974a7cca944db093d36241f22d663a P 5f96a2946e9b4ae9b7bb144ee282b912 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:04:41.009197 6963 tablet_replica.cc:330] T 33dd558b4b60478c97aad83f82b9f1e5 P 5f96a2946e9b4ae9b7bb144ee282b912: stopping tablet replica
I20240502 08:04:41.009693 6963 raft_consensus.cc:2229] T 33dd558b4b60478c97aad83f82b9f1e5 P 5f96a2946e9b4ae9b7bb144ee282b912 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:04:41.010021 6963 raft_consensus.cc:2258] T 33dd558b4b60478c97aad83f82b9f1e5 P 5f96a2946e9b4ae9b7bb144ee282b912 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:04:41.011723 6963 tablet_replica.cc:330] T d5f4dbf9e6f74bfc956ac9673786ccb5 P 5f96a2946e9b4ae9b7bb144ee282b912: stopping tablet replica
I20240502 08:04:41.012177 6963 raft_consensus.cc:2229] T d5f4dbf9e6f74bfc956ac9673786ccb5 P 5f96a2946e9b4ae9b7bb144ee282b912 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:04:41.012508 6963 raft_consensus.cc:2258] T d5f4dbf9e6f74bfc956ac9673786ccb5 P 5f96a2946e9b4ae9b7bb144ee282b912 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:04:41.014274 6963 tablet_replica.cc:330] T 20f890980e404e5ea6ec8995c97ed78c P 5f96a2946e9b4ae9b7bb144ee282b912: stopping tablet replica
I20240502 08:04:41.014708 6963 raft_consensus.cc:2229] T 20f890980e404e5ea6ec8995c97ed78c P 5f96a2946e9b4ae9b7bb144ee282b912 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:04:41.015034 6963 raft_consensus.cc:2258] T 20f890980e404e5ea6ec8995c97ed78c P 5f96a2946e9b4ae9b7bb144ee282b912 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:04:41.036134 6963 tablet_server.cc:194] TabletServer@127.6.204.195:0 shutdown complete.
I20240502 08:04:41.048764 6963 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:04:41.054767 12007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:04:41.056586 12010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:04:41.056581 12008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:04:41.057715 6963 server_base.cc:1034] running on GCE node
I20240502 08:04:41.058724 6963 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:04:41.059029 6963 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:04:41.059280 6963 hybrid_clock.cc:648] HybridClock initialized: now 1714637081059265 us; error 0 us; skew 500 ppm
I20240502 08:04:41.059882 6963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:04:41.062393 6963 webserver.cc:460] Webserver started at http://127.6.204.196:39251/ using document root <none> and password file <none>
I20240502 08:04:41.062935 6963 fs_manager.cc:359] Metadata directory not provided
I20240502 08:04:41.063215 6963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:04:41.063549 6963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:04:41.064707 6963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714636933625983-6963-0/minicluster-data/ts-3-root/instance:
uuid: "73ea4bf3ed49435c926de258601e43e0"
format_stamp: "Formatted at 2024-05-02 08:04:41 on dist-test-slave-32xs"
I20240502 08:04:41.069229 6963 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20240502 08:04:41.073026 12015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:04:41.073848 6963 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 08:04:41.074324 6963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714636933625983-6963-0/minicluster-data/ts-3-root
uuid: "73ea4bf3ed49435c926de258601e43e0"
format_stamp: "Formatted at 2024-05-02 08:04:41 on dist-test-slave-32xs"
I20240502 08:04:41.074721 6963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714636933625983-6963-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714636933625983-6963-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714636933625983-6963-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:04:41.086855 6963 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:04:41.087918 6963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:04:41.089462 6963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:04:41.091889 6963 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:04:41.092200 6963 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:04:41.092509 6963 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:04:41.092754 6963 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:04:41.130328 6963 rpc_server.cc:307] RPC server started. Bound to: 127.6.204.196:34479
I20240502 08:04:41.130419 12077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.204.196:34479 every 8 connection(s)
I20240502 08:04:41.143930 12078 heartbeater.cc:347] Connected to a master server at 127.6.204.254:38291
I20240502 08:04:41.144412 12078 heartbeater.cc:464] Registering TS with master...
I20240502 08:04:41.145187 12078 heartbeater.cc:511] Master 127.6.204.254:38291 requested a full tablet report, sending...
I20240502 08:04:41.147092 11684 ts_manager.cc:194] Registered new tserver with Master: 73ea4bf3ed49435c926de258601e43e0 (127.6.204.196:34479)
I20240502 08:04:41.148494 11684 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56400
W20240502 08:04:41.863176 11732 auto_rebalancer.cc:254] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.6.204.193:38381: connect: Connection refused (error 111)
W20240502 08:04:41.868846 11732 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.6.204.193:38381: connect: Connection refused (error 111)
W20240502 08:04:41.869275 11732 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.6.204.193:38381: connect: Connection refused (error 111)
W20240502 08:04:41.873299 11732 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.6.204.193:38381: connect: Connection refused (error 111)
W20240502 08:04:41.873665 11732 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.6.204.193:38381: connect: Connection refused (error 111)
W20240502 08:04:41.877663 11732 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.6.204.193:38381: connect: Connection refused (error 111)
W20240502 08:04:41.878033 11732 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.6.204.193:38381: connect: Connection refused (error 111)
I20240502 08:04:42.150988 12078 heartbeater.cc:503] Master 127.6.204.254:38291 was elected leader, sending a full tablet report...
W20240502 08:04:42.909133 11732 auto_rebalancer.cc:254] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.6.204.193:38381: connect: Connection refused (error 111)
W20240502 08:04:42.914856 11732 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.6.204.193:38381: connect: Connection refused (error 111)
W20240502 08:04:42.915108 11732 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.6.204.193:38381: connect: Connection refused (error 111)
W20240502 08:04:42.919159 11732 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.6.204.193:38381: connect: Connection refused (error 111)
W20240502 08:04:42.919453 11732 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.6.204.193:38381: connect: Connection refused (error 111)
W20240502 08:04:42.923630 11732 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.6.204.193:38381: connect: Connection refused (error 111)
W20240502 08:04:42.923877 11732 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.6.204.193:38381: connect: Connection refused (error 111)
W20240502 08:04:43.935426 11732 auto_rebalancer.cc:254] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.6.204.193:38381: connect: Connection refused (error 111)
W20240502 08:04:43.938773 11670 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.204.193:38381: connect: Connection refused (error 111) [suppressed 7 similar messages]
W20240502 08:04:43.941817 11732 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.6.204.193:38381: connect: Connection refused (error 111)
W20240502 08:04:43.942066 11732 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.6.204.193:38381: connect: Connection refused (error 111)
W20240502 08:04:43.946483 11732 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.6.204.193:38381: connect: Connection refused (error 111)
W20240502 08:04:43.946749 11732 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.6.204.193:38381: connect: Connection refused (error 111)
W20240502 08:04:43.951020 11732 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.6.204.193:38381: connect: Connection refused (error 111)
W20240502 08:04:43.951272 11732 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.6.204.193:38381: connect: Connection refused (error 111)
W20240502 08:04:44.964738 11732 auto_rebalancer.cc:254] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.6.204.193:38381: connect: Connection refused (error 111)
W20240502 08:04:44.970674 11732 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.6.204.193:38381: connect: Connection refused (error 111)
W20240502 08:04:44.970933 11732 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.6.204.193:38381: connect: Connection refused (error 111)
W20240502 08:04:44.974810 11732 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.6.204.193:38381: connect: Connection refused (error 111)
W20240502 08:04:44.975042 11732 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.6.204.193:38381: connect: Connection refused (error 111)
W20240502 08:04:44.978823 11732 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.6.204.193:38381: connect: Connection refused (error 111)
W20240502 08:04:44.979044 11732 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.6.204.193:38381: connect: Connection refused (error 111)
W20240502 08:04:45.981231 11732 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver c9eabd6e98644ebd8fcc4d9a9287cad3 not available for placement
W20240502 08:04:46.983361 11732 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver c9eabd6e98644ebd8fcc4d9a9287cad3 not available for placement
I20240502 08:04:47.170562 6963 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:04:47.179047 12095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:04:47.180665 12096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:04:47.181445 12098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:04:47.183758 6963 server_base.cc:1034] running on GCE node
I20240502 08:04:47.185433 6963 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:04:47.185633 6963 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:04:47.185768 6963 hybrid_clock.cc:648] HybridClock initialized: now 1714637087185758 us; error 0 us; skew 500 ppm
I20240502 08:04:47.186337 6963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:04:47.194326 6963 webserver.cc:460] Webserver started at http://127.6.204.193:42997/ using document root <none> and password file <none>
I20240502 08:04:47.194834 6963 fs_manager.cc:359] Metadata directory not provided
I20240502 08:04:47.194996 6963 fs_manager.cc:362] Using existing metadata directory in first data directory
I20240502 08:04:47.199213 6963 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20240502 08:04:47.202636 12103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:04:47.203544 6963 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20240502 08:04:47.203806 6963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714636933625983-6963-0/minicluster-data/ts-0-root
uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3"
format_stamp: "Formatted at 2024-05-02 08:04:38 on dist-test-slave-32xs"
I20240502 08:04:47.204176 6963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714636933625983-6963-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714636933625983-6963-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714636933625983-6963-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:04:47.227468 6963 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:04:47.228749 6963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:04:47.230304 6963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:04:47.234371 12110 ts_tablet_manager.cc:522] Loading tablet metadata (0/4 complete)
I20240502 08:04:47.257890 6963 ts_tablet_manager.cc:559] Loaded tablet metadata (4 total tablets, 4 live tablets)
I20240502 08:04:47.258256 6963 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.025s user 0.002s sys 0.000s
I20240502 08:04:47.258564 6963 ts_tablet_manager.cc:574] Registering tablets (0/4 complete)
I20240502 08:04:47.263772 12110 tablet_bootstrap.cc:492] T bf974a7cca944db093d36241f22d663a P c9eabd6e98644ebd8fcc4d9a9287cad3: Bootstrap starting.
I20240502 08:04:47.274469 6963 ts_tablet_manager.cc:590] Registered 4 tablets
I20240502 08:04:47.274725 6963 ts_tablet_manager.cc:569] Time spent register tablets: real 0.016s user 0.016s sys 0.000s
I20240502 08:04:47.276645 12110 tablet_bootstrap.cc:492] T bf974a7cca944db093d36241f22d663a P c9eabd6e98644ebd8fcc4d9a9287cad3: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 08:04:47.277511 12110 tablet_bootstrap.cc:492] T bf974a7cca944db093d36241f22d663a P c9eabd6e98644ebd8fcc4d9a9287cad3: Bootstrap complete.
I20240502 08:04:47.278074 12110 ts_tablet_manager.cc:1367] T bf974a7cca944db093d36241f22d663a P c9eabd6e98644ebd8fcc4d9a9287cad3: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.004s
I20240502 08:04:47.280403 12110 raft_consensus.cc:348] T bf974a7cca944db093d36241f22d663a P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
I20240502 08:04:47.281484 12110 raft_consensus.cc:729] T bf974a7cca944db093d36241f22d663a P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9eabd6e98644ebd8fcc4d9a9287cad3, State: Initialized, Role: FOLLOWER
I20240502 08:04:47.282181 12110 consensus_queue.cc:260] T bf974a7cca944db093d36241f22d663a P c9eabd6e98644ebd8fcc4d9a9287cad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
I20240502 08:04:47.285481 12110 ts_tablet_manager.cc:1398] T bf974a7cca944db093d36241f22d663a P c9eabd6e98644ebd8fcc4d9a9287cad3: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240502 08:04:47.286365 12110 tablet_bootstrap.cc:492] T 20f890980e404e5ea6ec8995c97ed78c P c9eabd6e98644ebd8fcc4d9a9287cad3: Bootstrap starting.
I20240502 08:04:47.301004 12110 tablet_bootstrap.cc:492] T 20f890980e404e5ea6ec8995c97ed78c P c9eabd6e98644ebd8fcc4d9a9287cad3: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 08:04:47.302250 12110 tablet_bootstrap.cc:492] T 20f890980e404e5ea6ec8995c97ed78c P c9eabd6e98644ebd8fcc4d9a9287cad3: Bootstrap complete.
I20240502 08:04:47.303017 12110 ts_tablet_manager.cc:1367] T 20f890980e404e5ea6ec8995c97ed78c P c9eabd6e98644ebd8fcc4d9a9287cad3: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.007s
I20240502 08:04:47.306355 12110 raft_consensus.cc:348] T 20f890980e404e5ea6ec8995c97ed78c P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
I20240502 08:04:47.307485 12110 raft_consensus.cc:729] T 20f890980e404e5ea6ec8995c97ed78c P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9eabd6e98644ebd8fcc4d9a9287cad3, State: Initialized, Role: FOLLOWER
I20240502 08:04:47.308426 12110 consensus_queue.cc:260] T 20f890980e404e5ea6ec8995c97ed78c P c9eabd6e98644ebd8fcc4d9a9287cad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
I20240502 08:04:47.311111 12110 ts_tablet_manager.cc:1398] T 20f890980e404e5ea6ec8995c97ed78c P c9eabd6e98644ebd8fcc4d9a9287cad3: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20240502 08:04:47.312147 12110 tablet_bootstrap.cc:492] T 33dd558b4b60478c97aad83f82b9f1e5 P c9eabd6e98644ebd8fcc4d9a9287cad3: Bootstrap starting.
I20240502 08:04:47.331240 12110 tablet_bootstrap.cc:492] T 33dd558b4b60478c97aad83f82b9f1e5 P c9eabd6e98644ebd8fcc4d9a9287cad3: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 08:04:47.331640 6963 rpc_server.cc:307] RPC server started. Bound to: 127.6.204.193:38381
I20240502 08:04:47.331794 12170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.204.193:38381 every 8 connection(s)
I20240502 08:04:47.332374 12110 tablet_bootstrap.cc:492] T 33dd558b4b60478c97aad83f82b9f1e5 P c9eabd6e98644ebd8fcc4d9a9287cad3: Bootstrap complete.
I20240502 08:04:47.333191 12110 ts_tablet_manager.cc:1367] T 33dd558b4b60478c97aad83f82b9f1e5 P c9eabd6e98644ebd8fcc4d9a9287cad3: Time spent bootstrapping tablet: real 0.021s user 0.014s sys 0.004s
I20240502 08:04:47.336303 12110 raft_consensus.cc:348] T 33dd558b4b60478c97aad83f82b9f1e5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
W20240502 08:04:47.336817 12172 tablet.cc:2369] T 20f890980e404e5ea6ec8995c97ed78c P c9eabd6e98644ebd8fcc4d9a9287cad3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 08:04:47.337422 12110 raft_consensus.cc:729] T 33dd558b4b60478c97aad83f82b9f1e5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9eabd6e98644ebd8fcc4d9a9287cad3, State: Initialized, Role: FOLLOWER
I20240502 08:04:47.337617 6963 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 08:04:47.338304 12110 consensus_queue.cc:260] T 33dd558b4b60478c97aad83f82b9f1e5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
I20240502 08:04:47.341955 12110 ts_tablet_manager.cc:1398] T 33dd558b4b60478c97aad83f82b9f1e5 P c9eabd6e98644ebd8fcc4d9a9287cad3: Time spent starting tablet: real 0.008s user 0.006s sys 0.004s
I20240502 08:04:47.343029 12110 tablet_bootstrap.cc:492] T d5f4dbf9e6f74bfc956ac9673786ccb5 P c9eabd6e98644ebd8fcc4d9a9287cad3: Bootstrap starting.
W20240502 08:04:47.346305 12175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:04:47.348302 12176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:04:47.361375 12178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:04:47.362075 6963 server_base.cc:1034] running on GCE node
I20240502 08:04:47.362475 12171 heartbeater.cc:347] Connected to a master server at 127.6.204.254:38291
I20240502 08:04:47.362946 12171 heartbeater.cc:464] Registering TS with master...
I20240502 08:04:47.363310 6963 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:04:47.363580 6963 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:04:47.363790 6963 hybrid_clock.cc:648] HybridClock initialized: now 1714637087363771 us; error 0 us; skew 500 ppm
I20240502 08:04:47.363898 12171 heartbeater.cc:511] Master 127.6.204.254:38291 requested a full tablet report, sending...
I20240502 08:04:47.364641 6963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:04:47.364864 12110 tablet_bootstrap.cc:492] T d5f4dbf9e6f74bfc956ac9673786ccb5 P c9eabd6e98644ebd8fcc4d9a9287cad3: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 08:04:47.365775 12110 tablet_bootstrap.cc:492] T d5f4dbf9e6f74bfc956ac9673786ccb5 P c9eabd6e98644ebd8fcc4d9a9287cad3: Bootstrap complete.
I20240502 08:04:47.366516 12110 ts_tablet_manager.cc:1367] T d5f4dbf9e6f74bfc956ac9673786ccb5 P c9eabd6e98644ebd8fcc4d9a9287cad3: Time spent bootstrapping tablet: real 0.024s user 0.016s sys 0.005s
I20240502 08:04:47.367796 6963 webserver.cc:460] Webserver started at http://127.6.204.194:35657/ using document root <none> and password file <none>
I20240502 08:04:47.368453 6963 fs_manager.cc:359] Metadata directory not provided
I20240502 08:04:47.368670 6963 fs_manager.cc:362] Using existing metadata directory in first data directory
I20240502 08:04:47.368996 11684 ts_manager.cc:194] Re-registered known tserver with Master: c9eabd6e98644ebd8fcc4d9a9287cad3 (127.6.204.193:38381)
I20240502 08:04:47.368682 12110 raft_consensus.cc:348] T d5f4dbf9e6f74bfc956ac9673786ccb5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
I20240502 08:04:47.369705 12110 raft_consensus.cc:729] T d5f4dbf9e6f74bfc956ac9673786ccb5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9eabd6e98644ebd8fcc4d9a9287cad3, State: Initialized, Role: FOLLOWER
I20240502 08:04:47.370666 12110 consensus_queue.cc:260] T d5f4dbf9e6f74bfc956ac9673786ccb5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
I20240502 08:04:47.372370 12110 ts_tablet_manager.cc:1398] T d5f4dbf9e6f74bfc956ac9673786ccb5 P c9eabd6e98644ebd8fcc4d9a9287cad3: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240502 08:04:47.373978 6963 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20240502 08:04:47.375912 11684 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56406
I20240502 08:04:47.378647 12184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:04:47.379370 12171 heartbeater.cc:503] Master 127.6.204.254:38291 was elected leader, sending a full tablet report...
I20240502 08:04:47.379576 6963 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240502 08:04:47.379922 6963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714636933625983-6963-0/minicluster-data/ts-1-root
uuid: "1871d0a968764692b1a09a51fd0dab79"
format_stamp: "Formatted at 2024-05-02 08:04:39 on dist-test-slave-32xs"
I20240502 08:04:47.380199 6963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714636933625983-6963-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714636933625983-6963-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714636933625983-6963-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:04:47.399632 6963 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:04:47.400872 6963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:04:47.402485 6963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:04:47.428534 6963 ts_tablet_manager.cc:559] Loaded tablet metadata (4 total tablets, 4 live tablets)
I20240502 08:04:47.428855 6963 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.024s user 0.001s sys 0.001s
I20240502 08:04:47.433821 12191 tablet_bootstrap.cc:492] T bf974a7cca944db093d36241f22d663a P 1871d0a968764692b1a09a51fd0dab79: Bootstrap starting.
I20240502 08:04:47.444603 6963 ts_tablet_manager.cc:590] Registered 4 tablets
I20240502 08:04:47.444881 6963 ts_tablet_manager.cc:569] Time spent register tablets: real 0.016s user 0.010s sys 0.003s
I20240502 08:04:47.445418 12191 tablet_bootstrap.cc:492] T bf974a7cca944db093d36241f22d663a P 1871d0a968764692b1a09a51fd0dab79: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:04:47.446264 12191 tablet_bootstrap.cc:492] T bf974a7cca944db093d36241f22d663a P 1871d0a968764692b1a09a51fd0dab79: Bootstrap complete.
I20240502 08:04:47.446857 12191 ts_tablet_manager.cc:1367] T bf974a7cca944db093d36241f22d663a P 1871d0a968764692b1a09a51fd0dab79: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.004s
I20240502 08:04:47.449033 12191 raft_consensus.cc:348] T bf974a7cca944db093d36241f22d663a P 1871d0a968764692b1a09a51fd0dab79 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
I20240502 08:04:47.449616 12191 raft_consensus.cc:729] T bf974a7cca944db093d36241f22d663a P 1871d0a968764692b1a09a51fd0dab79 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1871d0a968764692b1a09a51fd0dab79, State: Initialized, Role: FOLLOWER
I20240502 08:04:47.450363 12191 consensus_queue.cc:260] T bf974a7cca944db093d36241f22d663a P 1871d0a968764692b1a09a51fd0dab79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
I20240502 08:04:47.453261 12191 ts_tablet_manager.cc:1398] T bf974a7cca944db093d36241f22d663a P 1871d0a968764692b1a09a51fd0dab79: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240502 08:04:47.454417 12191 tablet_bootstrap.cc:492] T 20f890980e404e5ea6ec8995c97ed78c P 1871d0a968764692b1a09a51fd0dab79: Bootstrap starting.
I20240502 08:04:47.467574 12191 tablet_bootstrap.cc:492] T 20f890980e404e5ea6ec8995c97ed78c P 1871d0a968764692b1a09a51fd0dab79: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:04:47.468300 12191 tablet_bootstrap.cc:492] T 20f890980e404e5ea6ec8995c97ed78c P 1871d0a968764692b1a09a51fd0dab79: Bootstrap complete.
I20240502 08:04:47.468873 12191 ts_tablet_manager.cc:1367] T 20f890980e404e5ea6ec8995c97ed78c P 1871d0a968764692b1a09a51fd0dab79: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.004s
I20240502 08:04:47.470875 12191 raft_consensus.cc:348] T 20f890980e404e5ea6ec8995c97ed78c P 1871d0a968764692b1a09a51fd0dab79 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
I20240502 08:04:47.471421 12191 raft_consensus.cc:729] T 20f890980e404e5ea6ec8995c97ed78c P 1871d0a968764692b1a09a51fd0dab79 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1871d0a968764692b1a09a51fd0dab79, State: Initialized, Role: FOLLOWER
I20240502 08:04:47.472008 12191 consensus_queue.cc:260] T 20f890980e404e5ea6ec8995c97ed78c P 1871d0a968764692b1a09a51fd0dab79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
I20240502 08:04:47.473660 12191 ts_tablet_manager.cc:1398] T 20f890980e404e5ea6ec8995c97ed78c P 1871d0a968764692b1a09a51fd0dab79: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
I20240502 08:04:47.474364 12191 tablet_bootstrap.cc:492] T 33dd558b4b60478c97aad83f82b9f1e5 P 1871d0a968764692b1a09a51fd0dab79: Bootstrap starting.
I20240502 08:04:47.486809 12191 tablet_bootstrap.cc:492] T 33dd558b4b60478c97aad83f82b9f1e5 P 1871d0a968764692b1a09a51fd0dab79: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:04:47.487568 12191 tablet_bootstrap.cc:492] T 33dd558b4b60478c97aad83f82b9f1e5 P 1871d0a968764692b1a09a51fd0dab79: Bootstrap complete.
I20240502 08:04:47.488325 12191 ts_tablet_manager.cc:1367] T 33dd558b4b60478c97aad83f82b9f1e5 P 1871d0a968764692b1a09a51fd0dab79: Time spent bootstrapping tablet: real 0.014s user 0.013s sys 0.000s
I20240502 08:04:47.490361 12191 raft_consensus.cc:348] T 33dd558b4b60478c97aad83f82b9f1e5 P 1871d0a968764692b1a09a51fd0dab79 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
I20240502 08:04:47.491040 12191 raft_consensus.cc:729] T 33dd558b4b60478c97aad83f82b9f1e5 P 1871d0a968764692b1a09a51fd0dab79 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1871d0a968764692b1a09a51fd0dab79, State: Initialized, Role: FOLLOWER
I20240502 08:04:47.491628 12191 consensus_queue.cc:260] T 33dd558b4b60478c97aad83f82b9f1e5 P 1871d0a968764692b1a09a51fd0dab79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
I20240502 08:04:47.493258 12191 ts_tablet_manager.cc:1398] T 33dd558b4b60478c97aad83f82b9f1e5 P 1871d0a968764692b1a09a51fd0dab79: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 08:04:47.494035 12191 tablet_bootstrap.cc:492] T d5f4dbf9e6f74bfc956ac9673786ccb5 P 1871d0a968764692b1a09a51fd0dab79: Bootstrap starting.
I20240502 08:04:47.505056 12191 tablet_bootstrap.cc:492] T d5f4dbf9e6f74bfc956ac9673786ccb5 P 1871d0a968764692b1a09a51fd0dab79: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:04:47.505832 12191 tablet_bootstrap.cc:492] T d5f4dbf9e6f74bfc956ac9673786ccb5 P 1871d0a968764692b1a09a51fd0dab79: Bootstrap complete.
I20240502 08:04:47.506213 6963 rpc_server.cc:307] RPC server started. Bound to: 127.6.204.194:39105
I20240502 08:04:47.506481 12252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.204.194:39105 every 8 connection(s)
I20240502 08:04:47.506605 12191 ts_tablet_manager.cc:1367] T d5f4dbf9e6f74bfc956ac9673786ccb5 P 1871d0a968764692b1a09a51fd0dab79: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.003s
I20240502 08:04:47.508690 12191 raft_consensus.cc:348] T d5f4dbf9e6f74bfc956ac9673786ccb5 P 1871d0a968764692b1a09a51fd0dab79 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
I20240502 08:04:47.509248 12191 raft_consensus.cc:729] T d5f4dbf9e6f74bfc956ac9673786ccb5 P 1871d0a968764692b1a09a51fd0dab79 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1871d0a968764692b1a09a51fd0dab79, State: Initialized, Role: FOLLOWER
I20240502 08:04:47.510066 12191 consensus_queue.cc:260] T d5f4dbf9e6f74bfc956ac9673786ccb5 P 1871d0a968764692b1a09a51fd0dab79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
I20240502 08:04:47.510797 6963 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 08:04:47.511973 12191 ts_tablet_manager.cc:1398] T d5f4dbf9e6f74bfc956ac9673786ccb5 P 1871d0a968764692b1a09a51fd0dab79: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
W20240502 08:04:47.518725 12256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:04:47.521704 12257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:04:47.521873 12259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:04:47.523068 6963 server_base.cc:1034] running on GCE node
I20240502 08:04:47.524122 6963 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:04:47.524348 6963 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:04:47.524520 6963 hybrid_clock.cc:648] HybridClock initialized: now 1714637087524502 us; error 0 us; skew 500 ppm
I20240502 08:04:47.524802 12253 heartbeater.cc:347] Connected to a master server at 127.6.204.254:38291
I20240502 08:04:47.525130 6963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:04:47.525167 12253 heartbeater.cc:464] Registering TS with master...
I20240502 08:04:47.526222 12253 heartbeater.cc:511] Master 127.6.204.254:38291 requested a full tablet report, sending...
I20240502 08:04:47.528251 6963 webserver.cc:460] Webserver started at http://127.6.204.195:45809/ using document root <none> and password file <none>
I20240502 08:04:47.528749 6963 fs_manager.cc:359] Metadata directory not provided
I20240502 08:04:47.528940 6963 fs_manager.cc:362] Using existing metadata directory in first data directory
I20240502 08:04:47.529853 11684 ts_manager.cc:194] Re-registered known tserver with Master: 1871d0a968764692b1a09a51fd0dab79 (127.6.204.194:39105)
I20240502 08:04:47.532819 6963 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 08:04:47.534227 11684 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56408
I20240502 08:04:47.536384 12264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:04:47.537257 6963 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 08:04:47.537326 12253 heartbeater.cc:503] Master 127.6.204.254:38291 was elected leader, sending a full tablet report...
I20240502 08:04:47.537541 6963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714636933625983-6963-0/minicluster-data/ts-2-root
uuid: "5f96a2946e9b4ae9b7bb144ee282b912"
format_stamp: "Formatted at 2024-05-02 08:04:39 on dist-test-slave-32xs"
I20240502 08:04:47.537906 6963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714636933625983-6963-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714636933625983-6963-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714636933625983-6963-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:04:47.556874 6963 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:04:47.558269 6963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:04:47.559839 6963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:04:47.585150 6963 ts_tablet_manager.cc:559] Loaded tablet metadata (4 total tablets, 4 live tablets)
I20240502 08:04:47.585479 6963 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.023s user 0.001s sys 0.000s
I20240502 08:04:47.590294 12271 tablet_bootstrap.cc:492] T bf974a7cca944db093d36241f22d663a P 5f96a2946e9b4ae9b7bb144ee282b912: Bootstrap starting.
I20240502 08:04:47.603771 12271 tablet_bootstrap.cc:492] T bf974a7cca944db093d36241f22d663a P 5f96a2946e9b4ae9b7bb144ee282b912: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:04:47.604671 12271 tablet_bootstrap.cc:492] T bf974a7cca944db093d36241f22d663a P 5f96a2946e9b4ae9b7bb144ee282b912: Bootstrap complete.
I20240502 08:04:47.605396 12271 ts_tablet_manager.cc:1367] T bf974a7cca944db093d36241f22d663a P 5f96a2946e9b4ae9b7bb144ee282b912: Time spent bootstrapping tablet: real 0.015s user 0.011s sys 0.000s
I20240502 08:04:47.608060 6963 ts_tablet_manager.cc:590] Registered 4 tablets
I20240502 08:04:47.608377 6963 ts_tablet_manager.cc:569] Time spent register tablets: real 0.023s user 0.023s sys 0.000s
I20240502 08:04:47.607925 12271 raft_consensus.cc:348] T bf974a7cca944db093d36241f22d663a P 5f96a2946e9b4ae9b7bb144ee282b912 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
I20240502 08:04:47.608668 12271 raft_consensus.cc:729] T bf974a7cca944db093d36241f22d663a P 5f96a2946e9b4ae9b7bb144ee282b912 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f96a2946e9b4ae9b7bb144ee282b912, State: Initialized, Role: FOLLOWER
I20240502 08:04:47.609580 12271 consensus_queue.cc:260] T bf974a7cca944db093d36241f22d663a P 5f96a2946e9b4ae9b7bb144ee282b912 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
I20240502 08:04:47.612370 12271 ts_tablet_manager.cc:1398] T bf974a7cca944db093d36241f22d663a P 5f96a2946e9b4ae9b7bb144ee282b912: Time spent starting tablet: real 0.007s user 0.002s sys 0.007s
I20240502 08:04:47.613227 12271 tablet_bootstrap.cc:492] T 20f890980e404e5ea6ec8995c97ed78c P 5f96a2946e9b4ae9b7bb144ee282b912: Bootstrap starting.
I20240502 08:04:47.628772 12271 tablet_bootstrap.cc:492] T 20f890980e404e5ea6ec8995c97ed78c P 5f96a2946e9b4ae9b7bb144ee282b912: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:04:47.629525 12271 tablet_bootstrap.cc:492] T 20f890980e404e5ea6ec8995c97ed78c P 5f96a2946e9b4ae9b7bb144ee282b912: Bootstrap complete.
I20240502 08:04:47.630120 12271 ts_tablet_manager.cc:1367] T 20f890980e404e5ea6ec8995c97ed78c P 5f96a2946e9b4ae9b7bb144ee282b912: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.011s
I20240502 08:04:47.632123 12271 raft_consensus.cc:348] T 20f890980e404e5ea6ec8995c97ed78c P 5f96a2946e9b4ae9b7bb144ee282b912 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
I20240502 08:04:47.632700 12271 raft_consensus.cc:729] T 20f890980e404e5ea6ec8995c97ed78c P 5f96a2946e9b4ae9b7bb144ee282b912 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f96a2946e9b4ae9b7bb144ee282b912, State: Initialized, Role: FOLLOWER
I20240502 08:04:47.633407 12271 consensus_queue.cc:260] T 20f890980e404e5ea6ec8995c97ed78c P 5f96a2946e9b4ae9b7bb144ee282b912 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
I20240502 08:04:47.635113 12271 ts_tablet_manager.cc:1398] T 20f890980e404e5ea6ec8995c97ed78c P 5f96a2946e9b4ae9b7bb144ee282b912: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240502 08:04:47.635797 12271 tablet_bootstrap.cc:492] T 33dd558b4b60478c97aad83f82b9f1e5 P 5f96a2946e9b4ae9b7bb144ee282b912: Bootstrap starting.
I20240502 08:04:47.648339 12271 tablet_bootstrap.cc:492] T 33dd558b4b60478c97aad83f82b9f1e5 P 5f96a2946e9b4ae9b7bb144ee282b912: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:04:47.649173 12271 tablet_bootstrap.cc:492] T 33dd558b4b60478c97aad83f82b9f1e5 P 5f96a2946e9b4ae9b7bb144ee282b912: Bootstrap complete.
I20240502 08:04:47.649806 12271 ts_tablet_manager.cc:1367] T 33dd558b4b60478c97aad83f82b9f1e5 P 5f96a2946e9b4ae9b7bb144ee282b912: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.005s
I20240502 08:04:47.652040 12271 raft_consensus.cc:348] T 33dd558b4b60478c97aad83f82b9f1e5 P 5f96a2946e9b4ae9b7bb144ee282b912 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
I20240502 08:04:47.652668 12271 raft_consensus.cc:729] T 33dd558b4b60478c97aad83f82b9f1e5 P 5f96a2946e9b4ae9b7bb144ee282b912 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f96a2946e9b4ae9b7bb144ee282b912, State: Initialized, Role: FOLLOWER
I20240502 08:04:47.653385 12271 consensus_queue.cc:260] T 33dd558b4b60478c97aad83f82b9f1e5 P 5f96a2946e9b4ae9b7bb144ee282b912 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
I20240502 08:04:47.655241 12271 ts_tablet_manager.cc:1398] T 33dd558b4b60478c97aad83f82b9f1e5 P 5f96a2946e9b4ae9b7bb144ee282b912: Time spent starting tablet: real 0.005s user 0.001s sys 0.006s
I20240502 08:04:47.655931 12271 tablet_bootstrap.cc:492] T d5f4dbf9e6f74bfc956ac9673786ccb5 P 5f96a2946e9b4ae9b7bb144ee282b912: Bootstrap starting.
I20240502 08:04:47.669261 12271 tablet_bootstrap.cc:492] T d5f4dbf9e6f74bfc956ac9673786ccb5 P 5f96a2946e9b4ae9b7bb144ee282b912: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:04:47.670100 12271 tablet_bootstrap.cc:492] T d5f4dbf9e6f74bfc956ac9673786ccb5 P 5f96a2946e9b4ae9b7bb144ee282b912: Bootstrap complete.
I20240502 08:04:47.670774 12271 ts_tablet_manager.cc:1367] T d5f4dbf9e6f74bfc956ac9673786ccb5 P 5f96a2946e9b4ae9b7bb144ee282b912: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.004s
I20240502 08:04:47.672255 6963 rpc_server.cc:307] RPC server started. Bound to: 127.6.204.195:40469
I20240502 08:04:47.672492 12332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.204.195:40469 every 8 connection(s)
I20240502 08:04:47.673056 12271 raft_consensus.cc:348] T d5f4dbf9e6f74bfc956ac9673786ccb5 P 5f96a2946e9b4ae9b7bb144ee282b912 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
I20240502 08:04:47.673681 12271 raft_consensus.cc:729] T d5f4dbf9e6f74bfc956ac9673786ccb5 P 5f96a2946e9b4ae9b7bb144ee282b912 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f96a2946e9b4ae9b7bb144ee282b912, State: Initialized, Role: FOLLOWER
I20240502 08:04:47.674376 12271 consensus_queue.cc:260] T d5f4dbf9e6f74bfc956ac9673786ccb5 P 5f96a2946e9b4ae9b7bb144ee282b912 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
I20240502 08:04:47.676465 12271 ts_tablet_manager.cc:1398] T d5f4dbf9e6f74bfc956ac9673786ccb5 P 5f96a2946e9b4ae9b7bb144ee282b912: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240502 08:04:47.688308 12333 heartbeater.cc:347] Connected to a master server at 127.6.204.254:38291
I20240502 08:04:47.688830 12333 heartbeater.cc:464] Registering TS with master...
I20240502 08:04:47.689662 12333 heartbeater.cc:511] Master 127.6.204.254:38291 requested a full tablet report, sending...
I20240502 08:04:47.693220 11684 ts_manager.cc:194] Re-registered known tserver with Master: 5f96a2946e9b4ae9b7bb144ee282b912 (127.6.204.195:40469)
I20240502 08:04:47.697695 11684 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56420
I20240502 08:04:47.700322 12333 heartbeater.cc:503] Master 127.6.204.254:38291 was elected leader, sending a full tablet report...
W20240502 08:04:48.004135 11732 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Replica c9eabd6e98644ebd8fcc4d9a9287cad3 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } } }
W20240502 08:04:48.007905 11732 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 33dd558b4b60478c97aad83f82b9f1e5, TS 5f96a2946e9b4ae9b7bb144ee282b912 -> TS 73ea4bf3ed49435c926de258601e43e0 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
W20240502 08:04:48.008366 11732 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 33dd558b4b60478c97aad83f82b9f1e5, TS 5f96a2946e9b4ae9b7bb144ee282b912 -> TS 73ea4bf3ed49435c926de258601e43e0 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
W20240502 08:04:48.011623 11732 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet d5f4dbf9e6f74bfc956ac9673786ccb5, TS c9eabd6e98644ebd8fcc4d9a9287cad3 -> TS 73ea4bf3ed49435c926de258601e43e0 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
W20240502 08:04:48.011997 11732 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet d5f4dbf9e6f74bfc956ac9673786ccb5, TS c9eabd6e98644ebd8fcc4d9a9287cad3 -> TS 73ea4bf3ed49435c926de258601e43e0 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
W20240502 08:04:48.015247 11732 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 20f890980e404e5ea6ec8995c97ed78c, TS 1871d0a968764692b1a09a51fd0dab79 -> TS 73ea4bf3ed49435c926de258601e43e0 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
W20240502 08:04:48.015653 11732 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 20f890980e404e5ea6ec8995c97ed78c, TS 1871d0a968764692b1a09a51fd0dab79 -> TS 73ea4bf3ed49435c926de258601e43e0 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
I20240502 08:04:48.693310 12338 raft_consensus.cc:482] T bf974a7cca944db093d36241f22d663a P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:04:48.693886 12338 raft_consensus.cc:504] T bf974a7cca944db093d36241f22d663a P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
I20240502 08:04:48.696350 12338 leader_election.cc:288] T bf974a7cca944db093d36241f22d663a P c9eabd6e98644ebd8fcc4d9a9287cad3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1871d0a968764692b1a09a51fd0dab79 (127.6.204.194:39105), 5f96a2946e9b4ae9b7bb144ee282b912 (127.6.204.195:40469)
I20240502 08:04:48.710328 12306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf974a7cca944db093d36241f22d663a" candidate_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" is_pre_election: true
I20240502 08:04:48.711217 12226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf974a7cca944db093d36241f22d663a" candidate_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1871d0a968764692b1a09a51fd0dab79" is_pre_election: true
I20240502 08:04:48.711249 12306 raft_consensus.cc:2454] T bf974a7cca944db093d36241f22d663a P 5f96a2946e9b4ae9b7bb144ee282b912 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9eabd6e98644ebd8fcc4d9a9287cad3 in term 1.
I20240502 08:04:48.712066 12226 raft_consensus.cc:2454] T bf974a7cca944db093d36241f22d663a P 1871d0a968764692b1a09a51fd0dab79 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9eabd6e98644ebd8fcc4d9a9287cad3 in term 1.
I20240502 08:04:48.714020 12107 leader_election.cc:302] T bf974a7cca944db093d36241f22d663a P c9eabd6e98644ebd8fcc4d9a9287cad3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1871d0a968764692b1a09a51fd0dab79, c9eabd6e98644ebd8fcc4d9a9287cad3; no voters:
I20240502 08:04:48.715037 12338 raft_consensus.cc:2781] T bf974a7cca944db093d36241f22d663a P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:04:48.715351 12338 raft_consensus.cc:482] T bf974a7cca944db093d36241f22d663a P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:04:48.715620 12338 raft_consensus.cc:3037] T bf974a7cca944db093d36241f22d663a P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:04:48.721547 12338 raft_consensus.cc:504] T bf974a7cca944db093d36241f22d663a P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
I20240502 08:04:48.723160 12338 leader_election.cc:288] T bf974a7cca944db093d36241f22d663a P c9eabd6e98644ebd8fcc4d9a9287cad3 [CANDIDATE]: Term 2 election: Requested vote from peers 1871d0a968764692b1a09a51fd0dab79 (127.6.204.194:39105), 5f96a2946e9b4ae9b7bb144ee282b912 (127.6.204.195:40469)
I20240502 08:04:48.724164 12306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf974a7cca944db093d36241f22d663a" candidate_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5f96a2946e9b4ae9b7bb144ee282b912"
I20240502 08:04:48.724084 12226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf974a7cca944db093d36241f22d663a" candidate_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1871d0a968764692b1a09a51fd0dab79"
I20240502 08:04:48.724845 12306 raft_consensus.cc:3037] T bf974a7cca944db093d36241f22d663a P 5f96a2946e9b4ae9b7bb144ee282b912 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:04:48.724925 12226 raft_consensus.cc:3037] T bf974a7cca944db093d36241f22d663a P 1871d0a968764692b1a09a51fd0dab79 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:04:48.732400 12226 raft_consensus.cc:2454] T bf974a7cca944db093d36241f22d663a P 1871d0a968764692b1a09a51fd0dab79 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9eabd6e98644ebd8fcc4d9a9287cad3 in term 2.
I20240502 08:04:48.732400 12306 raft_consensus.cc:2454] T bf974a7cca944db093d36241f22d663a P 5f96a2946e9b4ae9b7bb144ee282b912 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9eabd6e98644ebd8fcc4d9a9287cad3 in term 2.
I20240502 08:04:48.733693 12107 leader_election.cc:302] T bf974a7cca944db093d36241f22d663a P c9eabd6e98644ebd8fcc4d9a9287cad3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1871d0a968764692b1a09a51fd0dab79, c9eabd6e98644ebd8fcc4d9a9287cad3; no voters:
I20240502 08:04:48.734390 12338 raft_consensus.cc:2781] T bf974a7cca944db093d36241f22d663a P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:04:48.735538 12338 raft_consensus.cc:686] T bf974a7cca944db093d36241f22d663a P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 2 LEADER]: Becoming Leader. State: Replica: c9eabd6e98644ebd8fcc4d9a9287cad3, State: Running, Role: LEADER
I20240502 08:04:48.736244 12338 consensus_queue.cc:237] T bf974a7cca944db093d36241f22d663a P c9eabd6e98644ebd8fcc4d9a9287cad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
I20240502 08:04:48.742899 11684 catalog_manager.cc:5484] T bf974a7cca944db093d36241f22d663a P c9eabd6e98644ebd8fcc4d9a9287cad3 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } health_report { overall_health: UNKNOWN } } }
I20240502 08:04:48.785796 12349 raft_consensus.cc:482] T 33dd558b4b60478c97aad83f82b9f1e5 P 1871d0a968764692b1a09a51fd0dab79 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:04:48.786329 12349 raft_consensus.cc:504] T 33dd558b4b60478c97aad83f82b9f1e5 P 1871d0a968764692b1a09a51fd0dab79 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
I20240502 08:04:48.788285 12349 leader_election.cc:288] T 33dd558b4b60478c97aad83f82b9f1e5 P 1871d0a968764692b1a09a51fd0dab79 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c9eabd6e98644ebd8fcc4d9a9287cad3 (127.6.204.193:38381), 5f96a2946e9b4ae9b7bb144ee282b912 (127.6.204.195:40469)
I20240502 08:04:48.800849 12306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33dd558b4b60478c97aad83f82b9f1e5" candidate_uuid: "1871d0a968764692b1a09a51fd0dab79" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" is_pre_election: true
I20240502 08:04:48.801502 12306 raft_consensus.cc:2454] T 33dd558b4b60478c97aad83f82b9f1e5 P 5f96a2946e9b4ae9b7bb144ee282b912 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1871d0a968764692b1a09a51fd0dab79 in term 1.
I20240502 08:04:48.801842 12145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33dd558b4b60478c97aad83f82b9f1e5" candidate_uuid: "1871d0a968764692b1a09a51fd0dab79" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" is_pre_election: true
I20240502 08:04:48.802520 12185 leader_election.cc:302] T 33dd558b4b60478c97aad83f82b9f1e5 P 1871d0a968764692b1a09a51fd0dab79 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1871d0a968764692b1a09a51fd0dab79, 5f96a2946e9b4ae9b7bb144ee282b912; no voters:
I20240502 08:04:48.802718 12145 raft_consensus.cc:2396] T 33dd558b4b60478c97aad83f82b9f1e5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1871d0a968764692b1a09a51fd0dab79 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20240502 08:04:48.803258 12349 raft_consensus.cc:2781] T 33dd558b4b60478c97aad83f82b9f1e5 P 1871d0a968764692b1a09a51fd0dab79 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:04:48.803557 12349 raft_consensus.cc:482] T 33dd558b4b60478c97aad83f82b9f1e5 P 1871d0a968764692b1a09a51fd0dab79 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:04:48.803836 12349 raft_consensus.cc:3037] T 33dd558b4b60478c97aad83f82b9f1e5 P 1871d0a968764692b1a09a51fd0dab79 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:04:48.809206 12349 raft_consensus.cc:504] T 33dd558b4b60478c97aad83f82b9f1e5 P 1871d0a968764692b1a09a51fd0dab79 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
I20240502 08:04:48.810720 12349 leader_election.cc:288] T 33dd558b4b60478c97aad83f82b9f1e5 P 1871d0a968764692b1a09a51fd0dab79 [CANDIDATE]: Term 2 election: Requested vote from peers c9eabd6e98644ebd8fcc4d9a9287cad3 (127.6.204.193:38381), 5f96a2946e9b4ae9b7bb144ee282b912 (127.6.204.195:40469)
I20240502 08:04:48.811640 12145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33dd558b4b60478c97aad83f82b9f1e5" candidate_uuid: "1871d0a968764692b1a09a51fd0dab79" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3"
I20240502 08:04:48.811810 12306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33dd558b4b60478c97aad83f82b9f1e5" candidate_uuid: "1871d0a968764692b1a09a51fd0dab79" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f96a2946e9b4ae9b7bb144ee282b912"
I20240502 08:04:48.812278 12145 raft_consensus.cc:3037] T 33dd558b4b60478c97aad83f82b9f1e5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:04:48.812348 12306 raft_consensus.cc:3037] T 33dd558b4b60478c97aad83f82b9f1e5 P 5f96a2946e9b4ae9b7bb144ee282b912 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:04:48.820086 12306 raft_consensus.cc:2454] T 33dd558b4b60478c97aad83f82b9f1e5 P 5f96a2946e9b4ae9b7bb144ee282b912 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1871d0a968764692b1a09a51fd0dab79 in term 2.
I20240502 08:04:48.820451 12145 raft_consensus.cc:2396] T 33dd558b4b60478c97aad83f82b9f1e5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1871d0a968764692b1a09a51fd0dab79 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20240502 08:04:48.821274 12185 leader_election.cc:302] T 33dd558b4b60478c97aad83f82b9f1e5 P 1871d0a968764692b1a09a51fd0dab79 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1871d0a968764692b1a09a51fd0dab79, 5f96a2946e9b4ae9b7bb144ee282b912; no voters:
I20240502 08:04:48.822001 12349 raft_consensus.cc:2781] T 33dd558b4b60478c97aad83f82b9f1e5 P 1871d0a968764692b1a09a51fd0dab79 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:04:48.823123 12349 raft_consensus.cc:686] T 33dd558b4b60478c97aad83f82b9f1e5 P 1871d0a968764692b1a09a51fd0dab79 [term 2 LEADER]: Becoming Leader. State: Replica: 1871d0a968764692b1a09a51fd0dab79, State: Running, Role: LEADER
I20240502 08:04:48.823776 12349 consensus_queue.cc:237] T 33dd558b4b60478c97aad83f82b9f1e5 P 1871d0a968764692b1a09a51fd0dab79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
I20240502 08:04:48.830679 11684 catalog_manager.cc:5484] T 33dd558b4b60478c97aad83f82b9f1e5 P 1871d0a968764692b1a09a51fd0dab79 reported cstate change: term changed from 1 to 2, leader changed from c9eabd6e98644ebd8fcc4d9a9287cad3 (127.6.204.193) to 1871d0a968764692b1a09a51fd0dab79 (127.6.204.194). New cstate: current_term: 2 leader_uuid: "1871d0a968764692b1a09a51fd0dab79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } health_report { overall_health: UNKNOWN } } }
I20240502 08:04:48.907260 12355 raft_consensus.cc:482] T 20f890980e404e5ea6ec8995c97ed78c P 5f96a2946e9b4ae9b7bb144ee282b912 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:04:48.907773 12355 raft_consensus.cc:504] T 20f890980e404e5ea6ec8995c97ed78c P 5f96a2946e9b4ae9b7bb144ee282b912 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
I20240502 08:04:48.909765 12355 leader_election.cc:288] T 20f890980e404e5ea6ec8995c97ed78c P 5f96a2946e9b4ae9b7bb144ee282b912 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c9eabd6e98644ebd8fcc4d9a9287cad3 (127.6.204.193:38381), 1871d0a968764692b1a09a51fd0dab79 (127.6.204.194:39105)
I20240502 08:04:48.922415 12226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20f890980e404e5ea6ec8995c97ed78c" candidate_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1871d0a968764692b1a09a51fd0dab79" is_pre_election: true
I20240502 08:04:48.922415 12145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20f890980e404e5ea6ec8995c97ed78c" candidate_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" is_pre_election: true
I20240502 08:04:48.923319 12226 raft_consensus.cc:2454] T 20f890980e404e5ea6ec8995c97ed78c P 1871d0a968764692b1a09a51fd0dab79 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f96a2946e9b4ae9b7bb144ee282b912 in term 1.
I20240502 08:04:48.923631 12145 raft_consensus.cc:2396] T 20f890980e404e5ea6ec8995c97ed78c P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f96a2946e9b4ae9b7bb144ee282b912 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20240502 08:04:48.924568 12268 leader_election.cc:302] T 20f890980e404e5ea6ec8995c97ed78c P 5f96a2946e9b4ae9b7bb144ee282b912 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1871d0a968764692b1a09a51fd0dab79, 5f96a2946e9b4ae9b7bb144ee282b912; no voters:
I20240502 08:04:48.925311 12355 raft_consensus.cc:2781] T 20f890980e404e5ea6ec8995c97ed78c P 5f96a2946e9b4ae9b7bb144ee282b912 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:04:48.925668 12355 raft_consensus.cc:482] T 20f890980e404e5ea6ec8995c97ed78c P 5f96a2946e9b4ae9b7bb144ee282b912 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:04:48.925956 12355 raft_consensus.cc:3037] T 20f890980e404e5ea6ec8995c97ed78c P 5f96a2946e9b4ae9b7bb144ee282b912 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:04:48.931195 12355 raft_consensus.cc:504] T 20f890980e404e5ea6ec8995c97ed78c P 5f96a2946e9b4ae9b7bb144ee282b912 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
I20240502 08:04:48.932698 12355 leader_election.cc:288] T 20f890980e404e5ea6ec8995c97ed78c P 5f96a2946e9b4ae9b7bb144ee282b912 [CANDIDATE]: Term 2 election: Requested vote from peers c9eabd6e98644ebd8fcc4d9a9287cad3 (127.6.204.193:38381), 1871d0a968764692b1a09a51fd0dab79 (127.6.204.194:39105)
I20240502 08:04:48.933653 12145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20f890980e404e5ea6ec8995c97ed78c" candidate_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3"
I20240502 08:04:48.933777 12226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20f890980e404e5ea6ec8995c97ed78c" candidate_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1871d0a968764692b1a09a51fd0dab79"
I20240502 08:04:48.934255 12145 raft_consensus.cc:3037] T 20f890980e404e5ea6ec8995c97ed78c P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:04:48.934461 12226 raft_consensus.cc:3037] T 20f890980e404e5ea6ec8995c97ed78c P 1871d0a968764692b1a09a51fd0dab79 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:04:48.940032 12145 raft_consensus.cc:2396] T 20f890980e404e5ea6ec8995c97ed78c P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 5f96a2946e9b4ae9b7bb144ee282b912 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20240502 08:04:48.940542 12226 raft_consensus.cc:2454] T 20f890980e404e5ea6ec8995c97ed78c P 1871d0a968764692b1a09a51fd0dab79 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f96a2946e9b4ae9b7bb144ee282b912 in term 2.
I20240502 08:04:48.941545 12268 leader_election.cc:302] T 20f890980e404e5ea6ec8995c97ed78c P 5f96a2946e9b4ae9b7bb144ee282b912 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1871d0a968764692b1a09a51fd0dab79, 5f96a2946e9b4ae9b7bb144ee282b912; no voters: c9eabd6e98644ebd8fcc4d9a9287cad3
I20240502 08:04:48.942250 12355 raft_consensus.cc:2781] T 20f890980e404e5ea6ec8995c97ed78c P 5f96a2946e9b4ae9b7bb144ee282b912 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:04:48.943312 12355 raft_consensus.cc:686] T 20f890980e404e5ea6ec8995c97ed78c P 5f96a2946e9b4ae9b7bb144ee282b912 [term 2 LEADER]: Becoming Leader. State: Replica: 5f96a2946e9b4ae9b7bb144ee282b912, State: Running, Role: LEADER
I20240502 08:04:48.943945 12355 consensus_queue.cc:237] T 20f890980e404e5ea6ec8995c97ed78c P 5f96a2946e9b4ae9b7bb144ee282b912 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
I20240502 08:04:48.950311 11684 catalog_manager.cc:5484] T 20f890980e404e5ea6ec8995c97ed78c P 5f96a2946e9b4ae9b7bb144ee282b912 reported cstate change: term changed from 1 to 2, leader changed from c9eabd6e98644ebd8fcc4d9a9287cad3 (127.6.204.193) to 5f96a2946e9b4ae9b7bb144ee282b912 (127.6.204.195). New cstate: current_term: 2 leader_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } health_report { overall_health: HEALTHY } } }
I20240502 08:04:49.007311 12338 raft_consensus.cc:482] T 33dd558b4b60478c97aad83f82b9f1e5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:04:49.007843 12338 raft_consensus.cc:504] T 33dd558b4b60478c97aad83f82b9f1e5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
I20240502 08:04:49.009467 12338 leader_election.cc:288] T 33dd558b4b60478c97aad83f82b9f1e5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1871d0a968764692b1a09a51fd0dab79 (127.6.204.194:39105), 5f96a2946e9b4ae9b7bb144ee282b912 (127.6.204.195:40469)
I20240502 08:04:49.010411 12226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33dd558b4b60478c97aad83f82b9f1e5" candidate_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1871d0a968764692b1a09a51fd0dab79" is_pre_election: true
I20240502 08:04:49.010540 12306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33dd558b4b60478c97aad83f82b9f1e5" candidate_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" is_pre_election: true
I20240502 08:04:49.011102 12306 raft_consensus.cc:2454] T 33dd558b4b60478c97aad83f82b9f1e5 P 5f96a2946e9b4ae9b7bb144ee282b912 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9eabd6e98644ebd8fcc4d9a9287cad3 in term 2.
I20240502 08:04:49.012085 12104 leader_election.cc:302] T 33dd558b4b60478c97aad83f82b9f1e5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f96a2946e9b4ae9b7bb144ee282b912, c9eabd6e98644ebd8fcc4d9a9287cad3; no voters: 1871d0a968764692b1a09a51fd0dab79
I20240502 08:04:49.012737 12338 raft_consensus.cc:2781] T 33dd558b4b60478c97aad83f82b9f1e5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 08:04:49.013087 12338 raft_consensus.cc:482] T 33dd558b4b60478c97aad83f82b9f1e5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:04:49.013340 12338 raft_consensus.cc:3037] T 33dd558b4b60478c97aad83f82b9f1e5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:04:49.019178 12338 raft_consensus.cc:504] T 33dd558b4b60478c97aad83f82b9f1e5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
I20240502 08:04:49.021160 12338 leader_election.cc:288] T 33dd558b4b60478c97aad83f82b9f1e5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [CANDIDATE]: Term 3 election: Requested vote from peers 1871d0a968764692b1a09a51fd0dab79 (127.6.204.194:39105), 5f96a2946e9b4ae9b7bb144ee282b912 (127.6.204.195:40469)
I20240502 08:04:49.022152 12226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33dd558b4b60478c97aad83f82b9f1e5" candidate_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1871d0a968764692b1a09a51fd0dab79"
I20240502 08:04:49.022326 12306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33dd558b4b60478c97aad83f82b9f1e5" candidate_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5f96a2946e9b4ae9b7bb144ee282b912"
I20240502 08:04:49.022987 12306 raft_consensus.cc:3037] T 33dd558b4b60478c97aad83f82b9f1e5 P 5f96a2946e9b4ae9b7bb144ee282b912 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:04:49.029161 12306 raft_consensus.cc:2454] T 33dd558b4b60478c97aad83f82b9f1e5 P 5f96a2946e9b4ae9b7bb144ee282b912 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9eabd6e98644ebd8fcc4d9a9287cad3 in term 3.
I20240502 08:04:49.030292 12104 leader_election.cc:302] T 33dd558b4b60478c97aad83f82b9f1e5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f96a2946e9b4ae9b7bb144ee282b912, c9eabd6e98644ebd8fcc4d9a9287cad3; no voters: 1871d0a968764692b1a09a51fd0dab79
I20240502 08:04:49.030934 12338 raft_consensus.cc:2781] T 33dd558b4b60478c97aad83f82b9f1e5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 08:04:49.031379 12338 raft_consensus.cc:686] T 33dd558b4b60478c97aad83f82b9f1e5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 3 LEADER]: Becoming Leader. State: Replica: c9eabd6e98644ebd8fcc4d9a9287cad3, State: Running, Role: LEADER
I20240502 08:04:49.032114 12338 consensus_queue.cc:237] T 33dd558b4b60478c97aad83f82b9f1e5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
W20240502 08:04:49.037708 11732 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Leader has not yet committed an operation in its own term
I20240502 08:04:49.039453 11684 catalog_manager.cc:5484] T 33dd558b4b60478c97aad83f82b9f1e5 P c9eabd6e98644ebd8fcc4d9a9287cad3 reported cstate change: term changed from 2 to 3, leader changed from 1871d0a968764692b1a09a51fd0dab79 (127.6.204.194) to c9eabd6e98644ebd8fcc4d9a9287cad3 (127.6.204.193). New cstate: current_term: 3 leader_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } health_report { overall_health: UNKNOWN } } }
W20240502 08:04:49.041752 11732 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 33dd558b4b60478c97aad83f82b9f1e5, TS 5f96a2946e9b4ae9b7bb144ee282b912 -> TS 73ea4bf3ed49435c926de258601e43e0 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
W20240502 08:04:49.042062 11732 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 33dd558b4b60478c97aad83f82b9f1e5, TS 5f96a2946e9b4ae9b7bb144ee282b912 -> TS 73ea4bf3ed49435c926de258601e43e0 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
W20240502 08:04:49.045358 11732 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet bf974a7cca944db093d36241f22d663a, TS 1871d0a968764692b1a09a51fd0dab79 -> TS 73ea4bf3ed49435c926de258601e43e0 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
W20240502 08:04:49.045564 11732 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet bf974a7cca944db093d36241f22d663a, TS 1871d0a968764692b1a09a51fd0dab79 -> TS 73ea4bf3ed49435c926de258601e43e0 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
W20240502 08:04:49.056552 11732 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 20f890980e404e5ea6ec8995c97ed78c, TS c9eabd6e98644ebd8fcc4d9a9287cad3 -> TS 73ea4bf3ed49435c926de258601e43e0 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
W20240502 08:04:49.056828 11732 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 20f890980e404e5ea6ec8995c97ed78c, TS c9eabd6e98644ebd8fcc4d9a9287cad3 -> TS 73ea4bf3ed49435c926de258601e43e0 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
I20240502 08:04:49.087116 12338 raft_consensus.cc:482] T d5f4dbf9e6f74bfc956ac9673786ccb5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:04:49.087612 12338 raft_consensus.cc:504] T d5f4dbf9e6f74bfc956ac9673786ccb5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
I20240502 08:04:49.089243 12338 leader_election.cc:288] T d5f4dbf9e6f74bfc956ac9673786ccb5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1871d0a968764692b1a09a51fd0dab79 (127.6.204.194:39105), 5f96a2946e9b4ae9b7bb144ee282b912 (127.6.204.195:40469)
I20240502 08:04:49.090185 12226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5f4dbf9e6f74bfc956ac9673786ccb5" candidate_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1871d0a968764692b1a09a51fd0dab79" is_pre_election: true
I20240502 08:04:49.090301 12306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5f4dbf9e6f74bfc956ac9673786ccb5" candidate_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" is_pre_election: true
I20240502 08:04:49.090763 12226 raft_consensus.cc:2454] T d5f4dbf9e6f74bfc956ac9673786ccb5 P 1871d0a968764692b1a09a51fd0dab79 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9eabd6e98644ebd8fcc4d9a9287cad3 in term 1.
I20240502 08:04:49.090843 12306 raft_consensus.cc:2454] T d5f4dbf9e6f74bfc956ac9673786ccb5 P 5f96a2946e9b4ae9b7bb144ee282b912 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9eabd6e98644ebd8fcc4d9a9287cad3 in term 1.
I20240502 08:04:49.091709 12107 leader_election.cc:302] T d5f4dbf9e6f74bfc956ac9673786ccb5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1871d0a968764692b1a09a51fd0dab79, c9eabd6e98644ebd8fcc4d9a9287cad3; no voters:
I20240502 08:04:49.092351 12338 raft_consensus.cc:2781] T d5f4dbf9e6f74bfc956ac9673786ccb5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:04:49.092690 12338 raft_consensus.cc:482] T d5f4dbf9e6f74bfc956ac9673786ccb5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:04:49.092914 12338 raft_consensus.cc:3037] T d5f4dbf9e6f74bfc956ac9673786ccb5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:04:49.097772 12338 raft_consensus.cc:504] T d5f4dbf9e6f74bfc956ac9673786ccb5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
I20240502 08:04:49.099375 12338 leader_election.cc:288] T d5f4dbf9e6f74bfc956ac9673786ccb5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [CANDIDATE]: Term 2 election: Requested vote from peers 1871d0a968764692b1a09a51fd0dab79 (127.6.204.194:39105), 5f96a2946e9b4ae9b7bb144ee282b912 (127.6.204.195:40469)
I20240502 08:04:49.100064 12226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5f4dbf9e6f74bfc956ac9673786ccb5" candidate_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1871d0a968764692b1a09a51fd0dab79"
I20240502 08:04:49.100315 12306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5f4dbf9e6f74bfc956ac9673786ccb5" candidate_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5f96a2946e9b4ae9b7bb144ee282b912"
I20240502 08:04:49.100557 12226 raft_consensus.cc:3037] T d5f4dbf9e6f74bfc956ac9673786ccb5 P 1871d0a968764692b1a09a51fd0dab79 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:04:49.100840 12306 raft_consensus.cc:3037] T d5f4dbf9e6f74bfc956ac9673786ccb5 P 5f96a2946e9b4ae9b7bb144ee282b912 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:04:49.105281 12226 raft_consensus.cc:2454] T d5f4dbf9e6f74bfc956ac9673786ccb5 P 1871d0a968764692b1a09a51fd0dab79 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9eabd6e98644ebd8fcc4d9a9287cad3 in term 2.
I20240502 08:04:49.105453 12306 raft_consensus.cc:2454] T d5f4dbf9e6f74bfc956ac9673786ccb5 P 5f96a2946e9b4ae9b7bb144ee282b912 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9eabd6e98644ebd8fcc4d9a9287cad3 in term 2.
I20240502 08:04:49.106362 12107 leader_election.cc:302] T d5f4dbf9e6f74bfc956ac9673786ccb5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1871d0a968764692b1a09a51fd0dab79, c9eabd6e98644ebd8fcc4d9a9287cad3; no voters:
I20240502 08:04:49.107044 12338 raft_consensus.cc:2781] T d5f4dbf9e6f74bfc956ac9673786ccb5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:04:49.107427 12338 raft_consensus.cc:686] T d5f4dbf9e6f74bfc956ac9673786ccb5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 2 LEADER]: Becoming Leader. State: Replica: c9eabd6e98644ebd8fcc4d9a9287cad3, State: Running, Role: LEADER
I20240502 08:04:49.108091 12338 consensus_queue.cc:237] T d5f4dbf9e6f74bfc956ac9673786ccb5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
I20240502 08:04:49.114599 11684 catalog_manager.cc:5484] T d5f4dbf9e6f74bfc956ac9673786ccb5 P c9eabd6e98644ebd8fcc4d9a9287cad3 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } health_report { overall_health: UNKNOWN } } }
I20240502 08:04:49.115370 12226 raft_consensus.cc:1261] T bf974a7cca944db093d36241f22d663a P 1871d0a968764692b1a09a51fd0dab79 [term 2 FOLLOWER]: Refusing update from remote peer c9eabd6e98644ebd8fcc4d9a9287cad3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:04:49.116716 12338 consensus_queue.cc:1035] T bf974a7cca944db093d36241f22d663a P c9eabd6e98644ebd8fcc4d9a9287cad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:04:49.130034 12306 raft_consensus.cc:1261] T bf974a7cca944db093d36241f22d663a P 5f96a2946e9b4ae9b7bb144ee282b912 [term 2 FOLLOWER]: Refusing update from remote peer c9eabd6e98644ebd8fcc4d9a9287cad3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:04:49.131582 12365 consensus_queue.cc:1035] T bf974a7cca944db093d36241f22d663a P c9eabd6e98644ebd8fcc4d9a9287cad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 08:04:49.147146 12365 raft_consensus.cc:482] T 20f890980e404e5ea6ec8995c97ed78c P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:04:49.147567 12365 raft_consensus.cc:504] T 20f890980e404e5ea6ec8995c97ed78c P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
I20240502 08:04:49.149077 12365 leader_election.cc:288] T 20f890980e404e5ea6ec8995c97ed78c P c9eabd6e98644ebd8fcc4d9a9287cad3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1871d0a968764692b1a09a51fd0dab79 (127.6.204.194:39105), 5f96a2946e9b4ae9b7bb144ee282b912 (127.6.204.195:40469)
I20240502 08:04:49.149868 12226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20f890980e404e5ea6ec8995c97ed78c" candidate_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1871d0a968764692b1a09a51fd0dab79" is_pre_election: true
I20240502 08:04:49.150102 12306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20f890980e404e5ea6ec8995c97ed78c" candidate_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" is_pre_election: true
I20240502 08:04:49.150703 12226 raft_consensus.cc:2454] T 20f890980e404e5ea6ec8995c97ed78c P 1871d0a968764692b1a09a51fd0dab79 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9eabd6e98644ebd8fcc4d9a9287cad3 in term 2.
I20240502 08:04:49.152280 12107 leader_election.cc:302] T 20f890980e404e5ea6ec8995c97ed78c P c9eabd6e98644ebd8fcc4d9a9287cad3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1871d0a968764692b1a09a51fd0dab79, c9eabd6e98644ebd8fcc4d9a9287cad3; no voters: 5f96a2946e9b4ae9b7bb144ee282b912
I20240502 08:04:49.152902 12365 raft_consensus.cc:2781] T 20f890980e404e5ea6ec8995c97ed78c P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 08:04:49.153228 12365 raft_consensus.cc:482] T 20f890980e404e5ea6ec8995c97ed78c P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:04:49.153456 12365 raft_consensus.cc:3037] T 20f890980e404e5ea6ec8995c97ed78c P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:04:49.158495 12365 raft_consensus.cc:504] T 20f890980e404e5ea6ec8995c97ed78c P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
I20240502 08:04:49.160066 12365 leader_election.cc:288] T 20f890980e404e5ea6ec8995c97ed78c P c9eabd6e98644ebd8fcc4d9a9287cad3 [CANDIDATE]: Term 3 election: Requested vote from peers 1871d0a968764692b1a09a51fd0dab79 (127.6.204.194:39105), 5f96a2946e9b4ae9b7bb144ee282b912 (127.6.204.195:40469)
I20240502 08:04:49.160931 12226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20f890980e404e5ea6ec8995c97ed78c" candidate_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1871d0a968764692b1a09a51fd0dab79"
I20240502 08:04:49.161180 12306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20f890980e404e5ea6ec8995c97ed78c" candidate_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5f96a2946e9b4ae9b7bb144ee282b912"
I20240502 08:04:49.161432 12226 raft_consensus.cc:3037] T 20f890980e404e5ea6ec8995c97ed78c P 1871d0a968764692b1a09a51fd0dab79 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:04:49.166214 12226 raft_consensus.cc:2454] T 20f890980e404e5ea6ec8995c97ed78c P 1871d0a968764692b1a09a51fd0dab79 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9eabd6e98644ebd8fcc4d9a9287cad3 in term 3.
I20240502 08:04:49.167193 12107 leader_election.cc:302] T 20f890980e404e5ea6ec8995c97ed78c P c9eabd6e98644ebd8fcc4d9a9287cad3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1871d0a968764692b1a09a51fd0dab79, c9eabd6e98644ebd8fcc4d9a9287cad3; no voters: 5f96a2946e9b4ae9b7bb144ee282b912
I20240502 08:04:49.167850 12365 raft_consensus.cc:2781] T 20f890980e404e5ea6ec8995c97ed78c P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 08:04:49.168277 12365 raft_consensus.cc:686] T 20f890980e404e5ea6ec8995c97ed78c P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 3 LEADER]: Becoming Leader. State: Replica: c9eabd6e98644ebd8fcc4d9a9287cad3, State: Running, Role: LEADER
I20240502 08:04:49.168861 12365 consensus_queue.cc:237] T 20f890980e404e5ea6ec8995c97ed78c P c9eabd6e98644ebd8fcc4d9a9287cad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
I20240502 08:04:49.175681 11684 catalog_manager.cc:5484] T 20f890980e404e5ea6ec8995c97ed78c P c9eabd6e98644ebd8fcc4d9a9287cad3 reported cstate change: term changed from 2 to 3, leader changed from 5f96a2946e9b4ae9b7bb144ee282b912 (127.6.204.195) to c9eabd6e98644ebd8fcc4d9a9287cad3 (127.6.204.193). New cstate: current_term: 3 leader_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } health_report { overall_health: UNKNOWN } } }
I20240502 08:04:49.255367 12145 raft_consensus.cc:1226] T 33dd558b4b60478c97aad83f82b9f1e5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 3 LEADER]: Rejecting Update request from peer 1871d0a968764692b1a09a51fd0dab79 for earlier term 2. Current term is 3. Ops: []
I20240502 08:04:49.256675 12349 consensus_queue.cc:1046] T 33dd558b4b60478c97aad83f82b9f1e5 P 1871d0a968764692b1a09a51fd0dab79 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:04:49.257351 12349 consensus_queue.cc:1225] T 33dd558b4b60478c97aad83f82b9f1e5 P 1871d0a968764692b1a09a51fd0dab79 [LEADER]: Peer c9eabd6e98644ebd8fcc4d9a9287cad3 log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20240502 08:04:49.257885 12353 raft_consensus.cc:3032] T 33dd558b4b60478c97aad83f82b9f1e5 P 1871d0a968764692b1a09a51fd0dab79 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 08:04:49.258172 12353 raft_consensus.cc:729] T 33dd558b4b60478c97aad83f82b9f1e5 P 1871d0a968764692b1a09a51fd0dab79 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1871d0a968764692b1a09a51fd0dab79, State: Running, Role: LEADER
I20240502 08:04:49.258781 12353 consensus_queue.cc:260] T 33dd558b4b60478c97aad83f82b9f1e5 P 1871d0a968764692b1a09a51fd0dab79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
I20240502 08:04:49.259680 12353 raft_consensus.cc:3037] T 33dd558b4b60478c97aad83f82b9f1e5 P 1871d0a968764692b1a09a51fd0dab79 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:04:49.331686 12145 raft_consensus.cc:1226] T 20f890980e404e5ea6ec8995c97ed78c P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 3 LEADER]: Rejecting Update request from peer 5f96a2946e9b4ae9b7bb144ee282b912 for earlier term 2. Current term is 3. Ops: []
I20240502 08:04:49.333012 12355 consensus_queue.cc:1046] T 20f890980e404e5ea6ec8995c97ed78c P 5f96a2946e9b4ae9b7bb144ee282b912 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:04:49.333709 12355 consensus_queue.cc:1225] T 20f890980e404e5ea6ec8995c97ed78c P 5f96a2946e9b4ae9b7bb144ee282b912 [LEADER]: Peer c9eabd6e98644ebd8fcc4d9a9287cad3 log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20240502 08:04:49.334249 12358 raft_consensus.cc:3032] T 20f890980e404e5ea6ec8995c97ed78c P 5f96a2946e9b4ae9b7bb144ee282b912 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 08:04:49.334529 12358 raft_consensus.cc:729] T 20f890980e404e5ea6ec8995c97ed78c P 5f96a2946e9b4ae9b7bb144ee282b912 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5f96a2946e9b4ae9b7bb144ee282b912, State: Running, Role: LEADER
I20240502 08:04:49.335218 12358 consensus_queue.cc:260] T 20f890980e404e5ea6ec8995c97ed78c P 5f96a2946e9b4ae9b7bb144ee282b912 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } }
I20240502 08:04:49.336118 12358 raft_consensus.cc:3037] T 20f890980e404e5ea6ec8995c97ed78c P 5f96a2946e9b4ae9b7bb144ee282b912 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:04:49.510829 12226 raft_consensus.cc:1261] T d5f4dbf9e6f74bfc956ac9673786ccb5 P 1871d0a968764692b1a09a51fd0dab79 [term 2 FOLLOWER]: Refusing update from remote peer c9eabd6e98644ebd8fcc4d9a9287cad3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:04:49.512079 12365 consensus_queue.cc:1035] T d5f4dbf9e6f74bfc956ac9673786ccb5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:04:49.523286 12306 raft_consensus.cc:1261] T d5f4dbf9e6f74bfc956ac9673786ccb5 P 5f96a2946e9b4ae9b7bb144ee282b912 [term 2 FOLLOWER]: Refusing update from remote peer c9eabd6e98644ebd8fcc4d9a9287cad3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:04:49.524781 12343 consensus_queue.cc:1035] T d5f4dbf9e6f74bfc956ac9673786ccb5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 08:04:49.583355 12226 raft_consensus.cc:1261] T 33dd558b4b60478c97aad83f82b9f1e5 P 1871d0a968764692b1a09a51fd0dab79 [term 3 FOLLOWER]: Refusing update from remote peer c9eabd6e98644ebd8fcc4d9a9287cad3: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240502 08:04:49.584506 12343 consensus_queue.cc:1035] T 33dd558b4b60478c97aad83f82b9f1e5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:04:49.585136 12343 consensus_queue.cc:1225] T 33dd558b4b60478c97aad83f82b9f1e5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [LEADER]: Peer 1871d0a968764692b1a09a51fd0dab79 log is divergent from this leader: its last log entry 2.1 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20240502 08:04:49.587366 12226 pending_rounds.cc:76] T 33dd558b4b60478c97aad83f82b9f1e5 P 1871d0a968764692b1a09a51fd0dab79: Aborting all ops after (but not including) 0
I20240502 08:04:49.587584 12226 pending_rounds.cc:97] T 33dd558b4b60478c97aad83f82b9f1e5 P 1871d0a968764692b1a09a51fd0dab79: Aborting uncommitted NO_OP operation due to leader change: 2.1
I20240502 08:04:49.587775 12226 raft_consensus.cc:2866] T 33dd558b4b60478c97aad83f82b9f1e5 P 1871d0a968764692b1a09a51fd0dab79 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 08:04:49.593653 12306 raft_consensus.cc:1261] T 33dd558b4b60478c97aad83f82b9f1e5 P 5f96a2946e9b4ae9b7bb144ee282b912 [term 3 FOLLOWER]: Refusing update from remote peer c9eabd6e98644ebd8fcc4d9a9287cad3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240502 08:04:49.595201 12365 consensus_queue.cc:1035] T 33dd558b4b60478c97aad83f82b9f1e5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:04:49.641814 12226 raft_consensus.cc:1261] T 20f890980e404e5ea6ec8995c97ed78c P 1871d0a968764692b1a09a51fd0dab79 [term 3 FOLLOWER]: Refusing update from remote peer c9eabd6e98644ebd8fcc4d9a9287cad3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240502 08:04:49.643051 12365 consensus_queue.cc:1035] T 20f890980e404e5ea6ec8995c97ed78c P c9eabd6e98644ebd8fcc4d9a9287cad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:04:49.654557 12306 raft_consensus.cc:1261] T 20f890980e404e5ea6ec8995c97ed78c P 5f96a2946e9b4ae9b7bb144ee282b912 [term 3 FOLLOWER]: Refusing update from remote peer c9eabd6e98644ebd8fcc4d9a9287cad3: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240502 08:04:49.655786 12343 consensus_queue.cc:1035] T 20f890980e404e5ea6ec8995c97ed78c P c9eabd6e98644ebd8fcc4d9a9287cad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:04:49.656586 12343 consensus_queue.cc:1225] T 20f890980e404e5ea6ec8995c97ed78c P c9eabd6e98644ebd8fcc4d9a9287cad3 [LEADER]: Peer 5f96a2946e9b4ae9b7bb144ee282b912 log is divergent from this leader: its last log entry 2.1 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20240502 08:04:49.659217 12306 pending_rounds.cc:76] T 20f890980e404e5ea6ec8995c97ed78c P 5f96a2946e9b4ae9b7bb144ee282b912: Aborting all ops after (but not including) 0
I20240502 08:04:49.659531 12306 pending_rounds.cc:97] T 20f890980e404e5ea6ec8995c97ed78c P 5f96a2946e9b4ae9b7bb144ee282b912: Aborting uncommitted NO_OP operation due to leader change: 2.1
I20240502 08:04:49.659790 12306 raft_consensus.cc:2866] T 20f890980e404e5ea6ec8995c97ed78c P 5f96a2946e9b4ae9b7bb144ee282b912 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 08:04:50.066633 12145 consensus_queue.cc:237] T bf974a7cca944db093d36241f22d663a P c9eabd6e98644ebd8fcc4d9a9287cad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } attrs { replace: true } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } } peers { permanent_uuid: "73ea4bf3ed49435c926de258601e43e0" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 34479 } attrs { promote: true } }
I20240502 08:04:50.073747 12306 raft_consensus.cc:1261] T bf974a7cca944db093d36241f22d663a P 5f96a2946e9b4ae9b7bb144ee282b912 [term 2 FOLLOWER]: Refusing update from remote peer c9eabd6e98644ebd8fcc4d9a9287cad3: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 08:04:50.075795 12343 consensus_queue.cc:1035] T bf974a7cca944db093d36241f22d663a P c9eabd6e98644ebd8fcc4d9a9287cad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:04:50.076738 12226 raft_consensus.cc:1261] T bf974a7cca944db093d36241f22d663a P 1871d0a968764692b1a09a51fd0dab79 [term 2 FOLLOWER]: Refusing update from remote peer c9eabd6e98644ebd8fcc4d9a9287cad3: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 08:04:50.078236 12343 consensus_queue.cc:1035] T bf974a7cca944db093d36241f22d663a P c9eabd6e98644ebd8fcc4d9a9287cad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:04:50.082670 12371 raft_consensus.cc:2932] T bf974a7cca944db093d36241f22d663a P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 73ea4bf3ed49435c926de258601e43e0 (127.6.204.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } attrs { replace: true } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } } peers { permanent_uuid: "73ea4bf3ed49435c926de258601e43e0" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 34479 } attrs { promote: true } } }
I20240502 08:04:50.084328 12306 raft_consensus.cc:2932] T bf974a7cca944db093d36241f22d663a P 5f96a2946e9b4ae9b7bb144ee282b912 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 73ea4bf3ed49435c926de258601e43e0 (127.6.204.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } attrs { replace: true } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } } peers { permanent_uuid: "73ea4bf3ed49435c926de258601e43e0" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 34479 } attrs { promote: true } } }
W20240502 08:04:50.087090 12106 consensus_peers.cc:485] T bf974a7cca944db093d36241f22d663a P c9eabd6e98644ebd8fcc4d9a9287cad3 -> Peer 73ea4bf3ed49435c926de258601e43e0 (127.6.204.196:34479): Couldn't send request to peer 73ea4bf3ed49435c926de258601e43e0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bf974a7cca944db093d36241f22d663a. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:04:50.097482 12145 consensus_queue.cc:237] T 33dd558b4b60478c97aad83f82b9f1e5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } attrs { replace: true } } peers { permanent_uuid: "73ea4bf3ed49435c926de258601e43e0" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 34479 } attrs { promote: true } }
I20240502 08:04:50.103855 12226 raft_consensus.cc:2932] T bf974a7cca944db093d36241f22d663a P 1871d0a968764692b1a09a51fd0dab79 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 73ea4bf3ed49435c926de258601e43e0 (127.6.204.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } attrs { replace: true } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } } peers { permanent_uuid: "73ea4bf3ed49435c926de258601e43e0" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 34479 } attrs { promote: true } } }
W20240502 08:04:50.108289 12106 consensus_peers.cc:485] T 33dd558b4b60478c97aad83f82b9f1e5 P c9eabd6e98644ebd8fcc4d9a9287cad3 -> Peer 73ea4bf3ed49435c926de258601e43e0 (127.6.204.196:34479): Couldn't send request to peer 73ea4bf3ed49435c926de258601e43e0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 33dd558b4b60478c97aad83f82b9f1e5. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:04:50.109190 12305 raft_consensus.cc:1261] T 33dd558b4b60478c97aad83f82b9f1e5 P 5f96a2946e9b4ae9b7bb144ee282b912 [term 3 FOLLOWER]: Refusing update from remote peer c9eabd6e98644ebd8fcc4d9a9287cad3: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 08:04:50.103380 11683 catalog_manager.cc:5484] T bf974a7cca944db093d36241f22d663a P c9eabd6e98644ebd8fcc4d9a9287cad3 reported cstate change: config changed from index -1 to 3, NON_VOTER 73ea4bf3ed49435c926de258601e43e0 (127.6.204.196) added. New cstate: current_term: 2 leader_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73ea4bf3ed49435c926de258601e43e0" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 34479 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:04:50.111819 12365 consensus_queue.cc:1035] T 33dd558b4b60478c97aad83f82b9f1e5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:04:50.120771 12225 raft_consensus.cc:1261] T 33dd558b4b60478c97aad83f82b9f1e5 P 1871d0a968764692b1a09a51fd0dab79 [term 3 FOLLOWER]: Refusing update from remote peer c9eabd6e98644ebd8fcc4d9a9287cad3: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 08:04:50.123471 12365 consensus_queue.cc:1035] T 33dd558b4b60478c97aad83f82b9f1e5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:04:50.124738 12377 raft_consensus.cc:2932] T 33dd558b4b60478c97aad83f82b9f1e5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 3 LEADER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 73ea4bf3ed49435c926de258601e43e0 (127.6.204.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } attrs { replace: true } } peers { permanent_uuid: "73ea4bf3ed49435c926de258601e43e0" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 34479 } attrs { promote: true } } }
I20240502 08:04:50.121986 12306 raft_consensus.cc:2932] T 33dd558b4b60478c97aad83f82b9f1e5 P 5f96a2946e9b4ae9b7bb144ee282b912 [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 73ea4bf3ed49435c926de258601e43e0 (127.6.204.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } attrs { replace: true } } peers { permanent_uuid: "73ea4bf3ed49435c926de258601e43e0" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 34479 } attrs { promote: true } } }
I20240502 08:04:50.127890 12226 raft_consensus.cc:2932] T 33dd558b4b60478c97aad83f82b9f1e5 P 1871d0a968764692b1a09a51fd0dab79 [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 73ea4bf3ed49435c926de258601e43e0 (127.6.204.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } attrs { replace: true } } peers { permanent_uuid: "73ea4bf3ed49435c926de258601e43e0" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 34479 } attrs { promote: true } } }
I20240502 08:04:50.141840 12145 consensus_queue.cc:237] T 20f890980e404e5ea6ec8995c97ed78c P c9eabd6e98644ebd8fcc4d9a9287cad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } attrs { replace: true } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } } peers { permanent_uuid: "73ea4bf3ed49435c926de258601e43e0" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 34479 } attrs { promote: true } }
I20240502 08:04:50.148725 12226 raft_consensus.cc:1261] T 20f890980e404e5ea6ec8995c97ed78c P 1871d0a968764692b1a09a51fd0dab79 [term 3 FOLLOWER]: Refusing update from remote peer c9eabd6e98644ebd8fcc4d9a9287cad3: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20240502 08:04:50.150333 12106 consensus_peers.cc:485] T 20f890980e404e5ea6ec8995c97ed78c P c9eabd6e98644ebd8fcc4d9a9287cad3 -> Peer 73ea4bf3ed49435c926de258601e43e0 (127.6.204.196:34479): Couldn't send request to peer 73ea4bf3ed49435c926de258601e43e0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 20f890980e404e5ea6ec8995c97ed78c. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:04:50.150540 12306 raft_consensus.cc:1261] T 20f890980e404e5ea6ec8995c97ed78c P 5f96a2946e9b4ae9b7bb144ee282b912 [term 3 FOLLOWER]: Refusing update from remote peer c9eabd6e98644ebd8fcc4d9a9287cad3: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 08:04:50.150266 11680 catalog_manager.cc:5484] T 33dd558b4b60478c97aad83f82b9f1e5 P 1871d0a968764692b1a09a51fd0dab79 reported cstate change: config changed from index -1 to 3, NON_VOTER 73ea4bf3ed49435c926de258601e43e0 (127.6.204.196) added. New cstate: current_term: 3 leader_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } attrs { replace: true } } peers { permanent_uuid: "73ea4bf3ed49435c926de258601e43e0" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 34479 } attrs { promote: true } } }
I20240502 08:04:50.152630 12365 consensus_queue.cc:1035] T 20f890980e404e5ea6ec8995c97ed78c P c9eabd6e98644ebd8fcc4d9a9287cad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:04:50.153656 12377 consensus_queue.cc:1035] T 20f890980e404e5ea6ec8995c97ed78c P c9eabd6e98644ebd8fcc4d9a9287cad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:04:50.159868 12343 raft_consensus.cc:2932] T 20f890980e404e5ea6ec8995c97ed78c P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 3 LEADER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 73ea4bf3ed49435c926de258601e43e0 (127.6.204.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } attrs { replace: true } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } } peers { permanent_uuid: "73ea4bf3ed49435c926de258601e43e0" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 34479 } attrs { promote: true } } }
I20240502 08:04:50.161494 12226 raft_consensus.cc:2932] T 20f890980e404e5ea6ec8995c97ed78c P 1871d0a968764692b1a09a51fd0dab79 [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 73ea4bf3ed49435c926de258601e43e0 (127.6.204.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } attrs { replace: true } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } } peers { permanent_uuid: "73ea4bf3ed49435c926de258601e43e0" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 34479 } attrs { promote: true } } }
I20240502 08:04:50.167647 12306 raft_consensus.cc:2932] T 20f890980e404e5ea6ec8995c97ed78c P 5f96a2946e9b4ae9b7bb144ee282b912 [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 73ea4bf3ed49435c926de258601e43e0 (127.6.204.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } attrs { replace: true } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } } peers { permanent_uuid: "73ea4bf3ed49435c926de258601e43e0" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 34479 } attrs { promote: true } } }
I20240502 08:04:50.176395 11684 catalog_manager.cc:5484] T 20f890980e404e5ea6ec8995c97ed78c P 1871d0a968764692b1a09a51fd0dab79 reported cstate change: config changed from index -1 to 3, NON_VOTER 73ea4bf3ed49435c926de258601e43e0 (127.6.204.196) added. New cstate: current_term: 3 leader_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } attrs { replace: true } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } } peers { permanent_uuid: "73ea4bf3ed49435c926de258601e43e0" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 34479 } attrs { promote: true } } }
I20240502 08:04:50.179587 12145 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bf974a7cca944db093d36241f22d663a"
dest_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:37700
I20240502 08:04:50.180263 12145 raft_consensus.cc:595] T bf974a7cca944db093d36241f22d663a P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 2 LEADER]: Received request to transfer leadership
I20240502 08:04:50.194902 12145 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bf974a7cca944db093d36241f22d663a"
dest_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:37700
I20240502 08:04:50.195379 12145 raft_consensus.cc:595] T bf974a7cca944db093d36241f22d663a P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 2 LEADER]: Received request to transfer leadership
W20240502 08:04:50.196338 11730 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: leadership transfer for bf974a7cca944db093d36241f22d663a already in progress [suppressed 5 similar messages]
I20240502 08:04:50.198514 12145 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bf974a7cca944db093d36241f22d663a"
dest_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:37700
I20240502 08:04:50.198933 12145 raft_consensus.cc:595] T bf974a7cca944db093d36241f22d663a P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 2 LEADER]: Received request to transfer leadership
W20240502 08:04:50.199908 11732 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for bf974a7cca944db093d36241f22d663a already in progress
W20240502 08:04:50.200170 11732 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for bf974a7cca944db093d36241f22d663a already in progress
I20240502 08:04:50.489271 12365 raft_consensus.cc:979] T bf974a7cca944db093d36241f22d663a P c9eabd6e98644ebd8fcc4d9a9287cad3: : Instructing follower 5f96a2946e9b4ae9b7bb144ee282b912 to start an election
I20240502 08:04:50.489722 12343 raft_consensus.cc:1067] T bf974a7cca944db093d36241f22d663a P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 2 LEADER]: Signalling peer 5f96a2946e9b4ae9b7bb144ee282b912 to start an election
I20240502 08:04:50.491505 12306 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "bf974a7cca944db093d36241f22d663a"
dest_uuid: "5f96a2946e9b4ae9b7bb144ee282b912"
from {username='slave'} at 127.0.0.1:41598
I20240502 08:04:50.492169 12306 raft_consensus.cc:482] T bf974a7cca944db093d36241f22d663a P 5f96a2946e9b4ae9b7bb144ee282b912 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:04:50.492596 12306 raft_consensus.cc:3037] T bf974a7cca944db093d36241f22d663a P 5f96a2946e9b4ae9b7bb144ee282b912 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:04:50.498767 12306 raft_consensus.cc:504] T bf974a7cca944db093d36241f22d663a P 5f96a2946e9b4ae9b7bb144ee282b912 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } attrs { replace: true } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } } peers { permanent_uuid: "73ea4bf3ed49435c926de258601e43e0" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 34479 } attrs { promote: true } }
I20240502 08:04:50.500788 12306 leader_election.cc:288] T bf974a7cca944db093d36241f22d663a P 5f96a2946e9b4ae9b7bb144ee282b912 [CANDIDATE]: Term 3 election: Requested vote from peers c9eabd6e98644ebd8fcc4d9a9287cad3 (127.6.204.193:38381), 1871d0a968764692b1a09a51fd0dab79 (127.6.204.194:39105)
I20240502 08:04:50.501572 12144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf974a7cca944db093d36241f22d663a" candidate_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: true dest_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3"
I20240502 08:04:50.502084 12144 raft_consensus.cc:3032] T bf974a7cca944db093d36241f22d663a P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 08:04:50.502390 12144 raft_consensus.cc:729] T bf974a7cca944db093d36241f22d663a P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c9eabd6e98644ebd8fcc4d9a9287cad3, State: Running, Role: LEADER
I20240502 08:04:50.503113 12144 consensus_queue.cc:260] T bf974a7cca944db093d36241f22d663a P c9eabd6e98644ebd8fcc4d9a9287cad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } attrs { replace: true } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } } peers { permanent_uuid: "73ea4bf3ed49435c926de258601e43e0" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 34479 } attrs { promote: true } }
I20240502 08:04:50.504416 12144 raft_consensus.cc:3037] T bf974a7cca944db093d36241f22d663a P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:04:50.510465 12226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf974a7cca944db093d36241f22d663a" candidate_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: true dest_uuid: "1871d0a968764692b1a09a51fd0dab79"
I20240502 08:04:50.511116 12226 raft_consensus.cc:3037] T bf974a7cca944db093d36241f22d663a P 1871d0a968764692b1a09a51fd0dab79 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:04:50.514425 12144 raft_consensus.cc:2454] T bf974a7cca944db093d36241f22d663a P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f96a2946e9b4ae9b7bb144ee282b912 in term 3.
I20240502 08:04:50.515568 12267 leader_election.cc:302] T bf974a7cca944db093d36241f22d663a P 5f96a2946e9b4ae9b7bb144ee282b912 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f96a2946e9b4ae9b7bb144ee282b912, c9eabd6e98644ebd8fcc4d9a9287cad3; no voters:
I20240502 08:04:50.516292 12358 raft_consensus.cc:2781] T bf974a7cca944db093d36241f22d663a P 5f96a2946e9b4ae9b7bb144ee282b912 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 08:04:50.516667 12226 raft_consensus.cc:2454] T bf974a7cca944db093d36241f22d663a P 1871d0a968764692b1a09a51fd0dab79 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f96a2946e9b4ae9b7bb144ee282b912 in term 3.
I20240502 08:04:50.518086 12358 raft_consensus.cc:686] T bf974a7cca944db093d36241f22d663a P 5f96a2946e9b4ae9b7bb144ee282b912 [term 3 LEADER]: Becoming Leader. State: Replica: 5f96a2946e9b4ae9b7bb144ee282b912, State: Running, Role: LEADER
I20240502 08:04:50.519016 12358 consensus_queue.cc:237] T bf974a7cca944db093d36241f22d663a P 5f96a2946e9b4ae9b7bb144ee282b912 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } attrs { replace: true } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } } peers { permanent_uuid: "73ea4bf3ed49435c926de258601e43e0" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 34479 } attrs { promote: true } }
I20240502 08:04:50.527011 11683 catalog_manager.cc:5484] T bf974a7cca944db093d36241f22d663a P 5f96a2946e9b4ae9b7bb144ee282b912 reported cstate change: term changed from 2 to 3, leader changed from c9eabd6e98644ebd8fcc4d9a9287cad3 (127.6.204.193) to 5f96a2946e9b4ae9b7bb144ee282b912 (127.6.204.195). New cstate: current_term: 3 leader_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73ea4bf3ed49435c926de258601e43e0" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 34479 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:04:50.532219 12383 ts_tablet_manager.cc:902] T 33dd558b4b60478c97aad83f82b9f1e5 P 73ea4bf3ed49435c926de258601e43e0: Initiating tablet copy from peer c9eabd6e98644ebd8fcc4d9a9287cad3 (127.6.204.193:38381)
I20240502 08:04:50.537284 12383 tablet_copy_client.cc:308] T 33dd558b4b60478c97aad83f82b9f1e5 P 73ea4bf3ed49435c926de258601e43e0: tablet copy: Beginning tablet copy session from remote peer at address 127.6.204.193:38381
I20240502 08:04:50.550684 12155 tablet_copy_service.cc:140] P c9eabd6e98644ebd8fcc4d9a9287cad3: Received BeginTabletCopySession request for tablet 33dd558b4b60478c97aad83f82b9f1e5 from peer 73ea4bf3ed49435c926de258601e43e0 ({username='slave'} at 127.0.0.1:47586)
I20240502 08:04:50.551321 12155 tablet_copy_service.cc:161] P c9eabd6e98644ebd8fcc4d9a9287cad3: Beginning new tablet copy session on tablet 33dd558b4b60478c97aad83f82b9f1e5 from peer 73ea4bf3ed49435c926de258601e43e0 at {username='slave'} at 127.0.0.1:47586: session id = 73ea4bf3ed49435c926de258601e43e0-33dd558b4b60478c97aad83f82b9f1e5
I20240502 08:04:50.558583 12155 tablet_copy_source_session.cc:205] T 33dd558b4b60478c97aad83f82b9f1e5 P c9eabd6e98644ebd8fcc4d9a9287cad3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:04:50.561410 12383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33dd558b4b60478c97aad83f82b9f1e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:50.574755 12383 tablet_copy_client.cc:783] T 33dd558b4b60478c97aad83f82b9f1e5 P 73ea4bf3ed49435c926de258601e43e0: tablet copy: Starting download of 0 data blocks...
I20240502 08:04:50.575333 12383 tablet_copy_client.cc:647] T 33dd558b4b60478c97aad83f82b9f1e5 P 73ea4bf3ed49435c926de258601e43e0: tablet copy: Starting download of 1 WAL segments...
I20240502 08:04:50.578465 12383 tablet_copy_client.cc:523] T 33dd558b4b60478c97aad83f82b9f1e5 P 73ea4bf3ed49435c926de258601e43e0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:04:50.585616 12383 tablet_bootstrap.cc:492] T 33dd558b4b60478c97aad83f82b9f1e5 P 73ea4bf3ed49435c926de258601e43e0: Bootstrap starting.
I20240502 08:04:50.601373 12383 tablet_bootstrap.cc:492] T 33dd558b4b60478c97aad83f82b9f1e5 P 73ea4bf3ed49435c926de258601e43e0: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:04:50.602152 12383 tablet_bootstrap.cc:492] T 33dd558b4b60478c97aad83f82b9f1e5 P 73ea4bf3ed49435c926de258601e43e0: Bootstrap complete.
I20240502 08:04:50.602754 12383 ts_tablet_manager.cc:1367] T 33dd558b4b60478c97aad83f82b9f1e5 P 73ea4bf3ed49435c926de258601e43e0: Time spent bootstrapping tablet: real 0.017s user 0.020s sys 0.000s
I20240502 08:04:50.604745 12383 raft_consensus.cc:348] T 33dd558b4b60478c97aad83f82b9f1e5 P 73ea4bf3ed49435c926de258601e43e0 [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } attrs { replace: true } } peers { permanent_uuid: "73ea4bf3ed49435c926de258601e43e0" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 34479 } attrs { promote: true } }
I20240502 08:04:50.605427 12383 raft_consensus.cc:729] T 33dd558b4b60478c97aad83f82b9f1e5 P 73ea4bf3ed49435c926de258601e43e0 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 73ea4bf3ed49435c926de258601e43e0, State: Initialized, Role: LEARNER
I20240502 08:04:50.605916 12383 consensus_queue.cc:260] T 33dd558b4b60478c97aad83f82b9f1e5 P 73ea4bf3ed49435c926de258601e43e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } attrs { replace: true } } peers { permanent_uuid: "73ea4bf3ed49435c926de258601e43e0" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 34479 } attrs { promote: true } }
I20240502 08:04:50.608595 12383 ts_tablet_manager.cc:1398] T 33dd558b4b60478c97aad83f82b9f1e5 P 73ea4bf3ed49435c926de258601e43e0: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240502 08:04:50.609993 12155 tablet_copy_service.cc:342] P c9eabd6e98644ebd8fcc4d9a9287cad3: Request end of tablet copy session 73ea4bf3ed49435c926de258601e43e0-33dd558b4b60478c97aad83f82b9f1e5 received from {username='slave'} at 127.0.0.1:47586
I20240502 08:04:50.610411 12155 tablet_copy_service.cc:433] P c9eabd6e98644ebd8fcc4d9a9287cad3: ending tablet copy session 73ea4bf3ed49435c926de258601e43e0-33dd558b4b60478c97aad83f82b9f1e5 on tablet 33dd558b4b60478c97aad83f82b9f1e5 with peer 73ea4bf3ed49435c926de258601e43e0
I20240502 08:04:50.680972 12383 ts_tablet_manager.cc:902] T 20f890980e404e5ea6ec8995c97ed78c P 73ea4bf3ed49435c926de258601e43e0: Initiating tablet copy from peer c9eabd6e98644ebd8fcc4d9a9287cad3 (127.6.204.193:38381)
I20240502 08:04:50.690522 12383 tablet_copy_client.cc:308] T 20f890980e404e5ea6ec8995c97ed78c P 73ea4bf3ed49435c926de258601e43e0: tablet copy: Beginning tablet copy session from remote peer at address 127.6.204.193:38381
I20240502 08:04:50.691886 12155 tablet_copy_service.cc:140] P c9eabd6e98644ebd8fcc4d9a9287cad3: Received BeginTabletCopySession request for tablet 20f890980e404e5ea6ec8995c97ed78c from peer 73ea4bf3ed49435c926de258601e43e0 ({username='slave'} at 127.0.0.1:47586)
I20240502 08:04:50.692420 12155 tablet_copy_service.cc:161] P c9eabd6e98644ebd8fcc4d9a9287cad3: Beginning new tablet copy session on tablet 20f890980e404e5ea6ec8995c97ed78c from peer 73ea4bf3ed49435c926de258601e43e0 at {username='slave'} at 127.0.0.1:47586: session id = 73ea4bf3ed49435c926de258601e43e0-20f890980e404e5ea6ec8995c97ed78c
I20240502 08:04:50.699478 12155 tablet_copy_source_session.cc:205] T 20f890980e404e5ea6ec8995c97ed78c P c9eabd6e98644ebd8fcc4d9a9287cad3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:04:50.702247 12383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20f890980e404e5ea6ec8995c97ed78c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:50.714267 12383 tablet_copy_client.cc:783] T 20f890980e404e5ea6ec8995c97ed78c P 73ea4bf3ed49435c926de258601e43e0: tablet copy: Starting download of 0 data blocks...
I20240502 08:04:50.714846 12383 tablet_copy_client.cc:647] T 20f890980e404e5ea6ec8995c97ed78c P 73ea4bf3ed49435c926de258601e43e0: tablet copy: Starting download of 1 WAL segments...
I20240502 08:04:50.717876 12383 tablet_copy_client.cc:523] T 20f890980e404e5ea6ec8995c97ed78c P 73ea4bf3ed49435c926de258601e43e0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:04:50.724885 12383 tablet_bootstrap.cc:492] T 20f890980e404e5ea6ec8995c97ed78c P 73ea4bf3ed49435c926de258601e43e0: Bootstrap starting.
I20240502 08:04:50.742074 12383 tablet_bootstrap.cc:492] T 20f890980e404e5ea6ec8995c97ed78c P 73ea4bf3ed49435c926de258601e43e0: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:04:50.742892 12383 tablet_bootstrap.cc:492] T 20f890980e404e5ea6ec8995c97ed78c P 73ea4bf3ed49435c926de258601e43e0: Bootstrap complete.
I20240502 08:04:50.743422 12383 ts_tablet_manager.cc:1367] T 20f890980e404e5ea6ec8995c97ed78c P 73ea4bf3ed49435c926de258601e43e0: Time spent bootstrapping tablet: real 0.019s user 0.016s sys 0.004s
I20240502 08:04:50.745601 12383 raft_consensus.cc:348] T 20f890980e404e5ea6ec8995c97ed78c P 73ea4bf3ed49435c926de258601e43e0 [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } attrs { replace: true } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } } peers { permanent_uuid: "73ea4bf3ed49435c926de258601e43e0" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 34479 } attrs { promote: true } }
I20240502 08:04:50.746241 12383 raft_consensus.cc:729] T 20f890980e404e5ea6ec8995c97ed78c P 73ea4bf3ed49435c926de258601e43e0 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 73ea4bf3ed49435c926de258601e43e0, State: Initialized, Role: LEARNER
I20240502 08:04:50.746802 12383 consensus_queue.cc:260] T 20f890980e404e5ea6ec8995c97ed78c P 73ea4bf3ed49435c926de258601e43e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } } peers { permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 } attrs { replace: true } } peers { permanent_uuid: "5f96a2946e9b4ae9b7bb144ee282b912" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 40469 } } peers { permanent_uuid: "73ea4bf3ed49435c926de258601e43e0" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 34479 } attrs { promote: true } }
I20240502 08:04:50.748811 12383 ts_tablet_manager.cc:1398] T 20f890980e404e5ea6ec8995c97ed78c P 73ea4bf3ed49435c926de258601e43e0: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240502 08:04:50.750357 12155 tablet_copy_service.cc:342] P c9eabd6e98644ebd8fcc4d9a9287cad3: Request end of tablet copy session 73ea4bf3ed49435c926de258601e43e0-20f890980e404e5ea6ec8995c97ed78c received from {username='slave'} at 127.0.0.1:47586
I20240502 08:04:50.750648 12155 tablet_copy_service.cc:433] P c9eabd6e98644ebd8fcc4d9a9287cad3: ending tablet copy session 73ea4bf3ed49435c926de258601e43e0-20f890980e404e5ea6ec8995c97ed78c on tablet 20f890980e404e5ea6ec8995c97ed78c with peer 73ea4bf3ed49435c926de258601e43e0
W20240502 08:04:51.020927 12267 consensus_peers.cc:485] T bf974a7cca944db093d36241f22d663a P 5f96a2946e9b4ae9b7bb144ee282b912 -> Peer 73ea4bf3ed49435c926de258601e43e0 (127.6.204.196:34479): Couldn't send request to peer 73ea4bf3ed49435c926de258601e43e0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bf974a7cca944db093d36241f22d663a. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:04:51.044195 12144 raft_consensus.cc:1261] T bf974a7cca944db093d36241f22d663a P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 3 FOLLOWER]: Refusing update from remote peer 5f96a2946e9b4ae9b7bb144ee282b912: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240502 08:04:51.045542 12358 consensus_queue.cc:1035] T bf974a7cca944db093d36241f22d663a P 5f96a2946e9b4ae9b7bb144ee282b912 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9eabd6e98644ebd8fcc4d9a9287cad3" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 38381 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:04:51.062554 12226 raft_consensus.cc:1261] T bf974a7cca944db093d36241f22d663a P 1871d0a968764692b1a09a51fd0dab79 [term 3 FOLLOWER]: Refusing update from remote peer 5f96a2946e9b4ae9b7bb144ee282b912: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240502 08:04:51.064020 12358 consensus_queue.cc:1035] T bf974a7cca944db093d36241f22d663a P 5f96a2946e9b4ae9b7bb144ee282b912 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1871d0a968764692b1a09a51fd0dab79" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240502 08:04:51.067083 12053 raft_consensus.cc:1203] T 33dd558b4b60478c97aad83f82b9f1e5 P 73ea4bf3ed49435c926de258601e43e0 [term 3 LEARNER]: Deduplicated request from leader. Original: 3.2->[3.3-3.3] Dedup: 3.3->[]
I20240502 08:04:51.097002 12053 raft_consensus.cc:1203] T 20f890980e404e5ea6ec8995c97ed78c P 73ea4bf3ed49435c926de258601e43e0 [term 3 LEARNER]: Deduplicated request from leader. Original: 3.2->[3.3-3.3] Dedup: 3.3->[]
I20240502 08:04:51.195235 6963 tablet_server.cc:178] TabletServer@127.6.204.193:38381 shutting down...
W20240502 08:04:51.200157 11732 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240502 08:04:51.200441 11732 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240502 08:04:51.205390 11732 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240502 08:04:51.205714 11732 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240502 08:04:51.217350 6963 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 08:04:51.218015 6963 tablet_replica.cc:330] stopping tablet replica
I20240502 08:04:51.218605 6963 raft_consensus.cc:2229] T 33dd558b4b60478c97aad83f82b9f1e5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 3 LEADER]: Raft consensus shutting down.
I20240502 08:04:51.219588 6963 raft_consensus.cc:2258] T 33dd558b4b60478c97aad83f82b9f1e5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 08:04:51.221376 6963 tablet_replica.cc:330] stopping tablet replica
I20240502 08:04:51.221882 6963 raft_consensus.cc:2229] T d5f4dbf9e6f74bfc956ac9673786ccb5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 2 LEADER]: Raft consensus shutting down.
I20240502 08:04:51.222616 6963 raft_consensus.cc:2258] T d5f4dbf9e6f74bfc956ac9673786ccb5 P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:04:51.224222 6963 tablet_replica.cc:330] stopping tablet replica
I20240502 08:04:51.224678 6963 raft_consensus.cc:2229] T 20f890980e404e5ea6ec8995c97ed78c P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 3 LEADER]: Raft consensus shutting down.
I20240502 08:04:51.225589 6963 raft_consensus.cc:2258] T 20f890980e404e5ea6ec8995c97ed78c P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 08:04:51.227377 6963 tablet_replica.cc:330] stopping tablet replica
I20240502 08:04:51.227955 6963 raft_consensus.cc:2229] T bf974a7cca944db093d36241f22d663a P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 08:04:51.228432 6963 raft_consensus.cc:2258] T bf974a7cca944db093d36241f22d663a P c9eabd6e98644ebd8fcc4d9a9287cad3 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 08:04:51.241869 6963 tablet_server.cc:194] TabletServer@127.6.204.193:38381 shutdown complete.
I20240502 08:04:51.253368 6963 tablet_server.cc:178] TabletServer@127.6.204.194:39105 shutting down...
I20240502 08:04:51.271734 6963 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 08:04:51.272332 6963 tablet_replica.cc:330] stopping tablet replica
I20240502 08:04:51.272910 6963 raft_consensus.cc:2229] T 33dd558b4b60478c97aad83f82b9f1e5 P 1871d0a968764692b1a09a51fd0dab79 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 08:04:51.273383 6963 raft_consensus.cc:2258] T 33dd558b4b60478c97aad83f82b9f1e5 P 1871d0a968764692b1a09a51fd0dab79 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 08:04:51.275204 6963 tablet_replica.cc:330] stopping tablet replica
I20240502 08:04:51.275704 6963 raft_consensus.cc:2229] T d5f4dbf9e6f74bfc956ac9673786ccb5 P 1871d0a968764692b1a09a51fd0dab79 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:04:51.276130 6963 raft_consensus.cc:2258] T d5f4dbf9e6f74bfc956ac9673786ccb5 P 1871d0a968764692b1a09a51fd0dab79 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:04:51.277735 6963 tablet_replica.cc:330] stopping tablet replica
I20240502 08:04:51.278252 6963 raft_consensus.cc:2229] T 20f890980e404e5ea6ec8995c97ed78c P 1871d0a968764692b1a09a51fd0dab79 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 08:04:51.278676 6963 raft_consensus.cc:2258] T 20f890980e404e5ea6ec8995c97ed78c P 1871d0a968764692b1a09a51fd0dab79 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 08:04:51.280331 6963 tablet_replica.cc:330] stopping tablet replica
I20240502 08:04:51.280774 6963 raft_consensus.cc:2229] T bf974a7cca944db093d36241f22d663a P 1871d0a968764692b1a09a51fd0dab79 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 08:04:51.281206 6963 raft_consensus.cc:2258] T bf974a7cca944db093d36241f22d663a P 1871d0a968764692b1a09a51fd0dab79 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 08:04:51.303123 6963 tablet_server.cc:194] TabletServer@127.6.204.194:39105 shutdown complete.
I20240502 08:04:51.313834 6963 tablet_server.cc:178] TabletServer@127.6.204.195:40469 shutting down...
I20240502 08:04:51.330044 6963 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 08:04:51.330765 6963 tablet_replica.cc:330] stopping tablet replica
I20240502 08:04:51.331315 6963 raft_consensus.cc:2229] T 33dd558b4b60478c97aad83f82b9f1e5 P 5f96a2946e9b4ae9b7bb144ee282b912 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 08:04:51.331807 6963 raft_consensus.cc:2258] T 33dd558b4b60478c97aad83f82b9f1e5 P 5f96a2946e9b4ae9b7bb144ee282b912 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 08:04:51.333647 6963 tablet_replica.cc:330] stopping tablet replica
I20240502 08:04:51.334204 6963 raft_consensus.cc:2229] T d5f4dbf9e6f74bfc956ac9673786ccb5 P 5f96a2946e9b4ae9b7bb144ee282b912 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:04:51.334617 6963 raft_consensus.cc:2258] T d5f4dbf9e6f74bfc956ac9673786ccb5 P 5f96a2946e9b4ae9b7bb144ee282b912 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:04:51.336208 6963 tablet_replica.cc:330] stopping tablet replica
I20240502 08:04:51.336653 6963 raft_consensus.cc:2229] T 20f890980e404e5ea6ec8995c97ed78c P 5f96a2946e9b4ae9b7bb144ee282b912 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 08:04:51.337085 6963 raft_consensus.cc:2258] T 20f890980e404e5ea6ec8995c97ed78c P 5f96a2946e9b4ae9b7bb144ee282b912 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 08:04:51.338822 6963 tablet_replica.cc:330] stopping tablet replica
I20240502 08:04:51.339248 6963 raft_consensus.cc:2229] T bf974a7cca944db093d36241f22d663a P 5f96a2946e9b4ae9b7bb144ee282b912 [term 3 LEADER]: Raft consensus shutting down.
I20240502 08:04:51.340276 6963 raft_consensus.cc:2258] T bf974a7cca944db093d36241f22d663a P 5f96a2946e9b4ae9b7bb144ee282b912 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 08:04:51.363425 6963 tablet_server.cc:194] TabletServer@127.6.204.195:40469 shutdown complete.
I20240502 08:04:51.374243 6963 tablet_server.cc:178] TabletServer@127.6.204.196:0 shutting down...
I20240502 08:04:51.393075 6963 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 08:04:51.393707 6963 tablet_replica.cc:330] T 20f890980e404e5ea6ec8995c97ed78c P 73ea4bf3ed49435c926de258601e43e0: stopping tablet replica
I20240502 08:04:51.394274 6963 raft_consensus.cc:2229] T 20f890980e404e5ea6ec8995c97ed78c P 73ea4bf3ed49435c926de258601e43e0 [term 3 LEARNER]: Raft consensus shutting down.
I20240502 08:04:51.394661 6963 raft_consensus.cc:2258] T 20f890980e404e5ea6ec8995c97ed78c P 73ea4bf3ed49435c926de258601e43e0 [term 3 LEARNER]: Raft consensus is shut down!
I20240502 08:04:51.396364 6963 tablet_replica.cc:330] T 33dd558b4b60478c97aad83f82b9f1e5 P 73ea4bf3ed49435c926de258601e43e0: stopping tablet replica
I20240502 08:04:51.396795 6963 raft_consensus.cc:2229] T 33dd558b4b60478c97aad83f82b9f1e5 P 73ea4bf3ed49435c926de258601e43e0 [term 3 LEARNER]: Raft consensus shutting down.
I20240502 08:04:51.397154 6963 raft_consensus.cc:2258] T 33dd558b4b60478c97aad83f82b9f1e5 P 73ea4bf3ed49435c926de258601e43e0 [term 3 LEARNER]: Raft consensus is shut down!
I20240502 08:04:51.416353 6963 tablet_server.cc:194] TabletServer@127.6.204.196:0 shutdown complete.
I20240502 08:04:51.426781 6963 master.cc:537] Master@127.6.204.254:38291 shutting down...
I20240502 08:04:51.444860 6963 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 7b1d8ba2ea8b49f7a7b78e8019c8c0bf [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:04:51.445436 6963 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 7b1d8ba2ea8b49f7a7b78e8019c8c0bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:04:51.445721 6963 tablet_replica.cc:330] T 00000000000000000000000000000000 P 7b1d8ba2ea8b49f7a7b78e8019c8c0bf: stopping tablet replica
I20240502 08:04:51.464469 6963 master.cc:552] Master@127.6.204.254:38291 shutdown complete.
[ OK ] AutoRebalancerTest.TestHandlingFailedTservers (12791 ms)
[ RUN ] AutoRebalancerTest.TestDeletedTables
I20240502 08:04:51.499783 6963 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.6.204.254:32811
I20240502 08:04:51.500818 6963 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:04:51.506029 12407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:04:51.506428 12408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:04:51.507448 12410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:04:51.508159 6963 server_base.cc:1034] running on GCE node
I20240502 08:04:51.510023 6963 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:04:51.510360 6963 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:04:51.510514 6963 hybrid_clock.cc:648] HybridClock initialized: now 1714637091510500 us; error 0 us; skew 500 ppm
I20240502 08:04:51.511029 6963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:04:51.513350 6963 webserver.cc:460] Webserver started at http://127.6.204.254:37577/ using document root <none> and password file <none>
I20240502 08:04:51.513793 6963 fs_manager.cc:359] Metadata directory not provided
I20240502 08:04:51.513960 6963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:04:51.514236 6963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:04:51.515340 6963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714636933625983-6963-0/minicluster-data/master-0-root/instance:
uuid: "b4c502a6b1fa4eb79c635850e5fa3883"
format_stamp: "Formatted at 2024-05-02 08:04:51 on dist-test-slave-32xs"
I20240502 08:04:51.519804 6963 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20240502 08:04:51.523092 12415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:04:51.523854 6963 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 08:04:51.524132 6963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714636933625983-6963-0/minicluster-data/master-0-root
uuid: "b4c502a6b1fa4eb79c635850e5fa3883"
format_stamp: "Formatted at 2024-05-02 08:04:51 on dist-test-slave-32xs"
I20240502 08:04:51.524397 6963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714636933625983-6963-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714636933625983-6963-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714636933625983-6963-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:04:51.545848 6963 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:04:51.547068 6963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:04:51.583577 6963 rpc_server.cc:307] RPC server started. Bound to: 127.6.204.254:32811
I20240502 08:04:51.583696 12466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.204.254:32811 every 8 connection(s)
I20240502 08:04:51.587280 12467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:51.597563 12467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4c502a6b1fa4eb79c635850e5fa3883: Bootstrap starting.
I20240502 08:04:51.601716 12467 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4c502a6b1fa4eb79c635850e5fa3883: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:51.605716 12467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4c502a6b1fa4eb79c635850e5fa3883: No bootstrap required, opened a new log
I20240502 08:04:51.607642 12467 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4c502a6b1fa4eb79c635850e5fa3883 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4c502a6b1fa4eb79c635850e5fa3883" member_type: VOTER }
I20240502 08:04:51.608021 12467 raft_consensus.cc:374] T 00000000000000000000000000000000 P b4c502a6b1fa4eb79c635850e5fa3883 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:51.608191 12467 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4c502a6b1fa4eb79c635850e5fa3883 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4c502a6b1fa4eb79c635850e5fa3883, State: Initialized, Role: FOLLOWER
I20240502 08:04:51.608691 12467 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4c502a6b1fa4eb79c635850e5fa3883 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4c502a6b1fa4eb79c635850e5fa3883" member_type: VOTER }
I20240502 08:04:51.609145 12467 raft_consensus.cc:388] T 00000000000000000000000000000000 P b4c502a6b1fa4eb79c635850e5fa3883 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:04:51.609329 12467 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4c502a6b1fa4eb79c635850e5fa3883 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:04:51.609537 12467 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4c502a6b1fa4eb79c635850e5fa3883 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:51.613883 12467 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4c502a6b1fa4eb79c635850e5fa3883 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4c502a6b1fa4eb79c635850e5fa3883" member_type: VOTER }
I20240502 08:04:51.614406 12467 leader_election.cc:302] T 00000000000000000000000000000000 P b4c502a6b1fa4eb79c635850e5fa3883 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4c502a6b1fa4eb79c635850e5fa3883; no voters:
I20240502 08:04:51.615523 12467 leader_election.cc:288] T 00000000000000000000000000000000 P b4c502a6b1fa4eb79c635850e5fa3883 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:04:51.615917 12470 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4c502a6b1fa4eb79c635850e5fa3883 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:04:51.617236 12470 raft_consensus.cc:686] T 00000000000000000000000000000000 P b4c502a6b1fa4eb79c635850e5fa3883 [term 1 LEADER]: Becoming Leader. State: Replica: b4c502a6b1fa4eb79c635850e5fa3883, State: Running, Role: LEADER
I20240502 08:04:51.617877 12470 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4c502a6b1fa4eb79c635850e5fa3883 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4c502a6b1fa4eb79c635850e5fa3883" member_type: VOTER }
I20240502 08:04:51.618678 12467 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4c502a6b1fa4eb79c635850e5fa3883 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 08:04:51.621057 12472 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4c502a6b1fa4eb79c635850e5fa3883 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4c502a6b1fa4eb79c635850e5fa3883. Latest consensus state: current_term: 1 leader_uuid: "b4c502a6b1fa4eb79c635850e5fa3883" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4c502a6b1fa4eb79c635850e5fa3883" member_type: VOTER } }
I20240502 08:04:51.621057 12471 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4c502a6b1fa4eb79c635850e5fa3883 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b4c502a6b1fa4eb79c635850e5fa3883" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4c502a6b1fa4eb79c635850e5fa3883" member_type: VOTER } }
I20240502 08:04:51.621770 12472 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4c502a6b1fa4eb79c635850e5fa3883 [sys.catalog]: This master's current role is: LEADER
I20240502 08:04:51.621944 12471 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4c502a6b1fa4eb79c635850e5fa3883 [sys.catalog]: This master's current role is: LEADER
I20240502 08:04:51.626992 12475 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 08:04:51.631688 12475 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 08:04:51.634884 6963 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 08:04:51.640336 12475 catalog_manager.cc:1347] Generated new cluster ID: 5cc1469e910b4609bf07054b59049306
I20240502 08:04:51.640611 12475 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 08:04:51.672622 12475 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 08:04:51.673862 12475 catalog_manager.cc:1504] Loading token signing keys...
I20240502 08:04:51.687615 12475 catalog_manager.cc:5857] T 00000000000000000000000000000000 P b4c502a6b1fa4eb79c635850e5fa3883: Generated new TSK 0
I20240502 08:04:51.688212 12475 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 08:04:51.702332 6963 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:04:51.707916 12488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:04:51.709522 12489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:04:51.711180 6963 server_base.cc:1034] running on GCE node
W20240502 08:04:51.711232 12491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:04:51.712316 6963 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:04:51.712518 6963 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:04:51.712649 6963 hybrid_clock.cc:648] HybridClock initialized: now 1714637091712638 us; error 0 us; skew 500 ppm
I20240502 08:04:51.713120 6963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:04:51.715433 6963 webserver.cc:460] Webserver started at http://127.6.204.193:37837/ using document root <none> and password file <none>
I20240502 08:04:51.715847 6963 fs_manager.cc:359] Metadata directory not provided
I20240502 08:04:51.716003 6963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:04:51.716218 6963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:04:51.717276 6963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714636933625983-6963-0/minicluster-data/ts-0-root/instance:
uuid: "7e739706a3cb4269b73b3dc63e28ccfe"
format_stamp: "Formatted at 2024-05-02 08:04:51 on dist-test-slave-32xs"
I20240502 08:04:51.721943 6963 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20240502 08:04:51.725184 12496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:04:51.725917 6963 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 08:04:51.726226 6963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714636933625983-6963-0/minicluster-data/ts-0-root
uuid: "7e739706a3cb4269b73b3dc63e28ccfe"
format_stamp: "Formatted at 2024-05-02 08:04:51 on dist-test-slave-32xs"
I20240502 08:04:51.726531 6963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714636933625983-6963-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714636933625983-6963-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714636933625983-6963-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:04:51.751365 6963 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:04:51.752542 6963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:04:51.754034 6963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:04:51.756400 6963 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:04:51.756600 6963 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:04:51.756841 6963 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:04:51.756997 6963 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:04:51.797955 6963 rpc_server.cc:307] RPC server started. Bound to: 127.6.204.193:32777
I20240502 08:04:51.798034 12558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.204.193:32777 every 8 connection(s)
I20240502 08:04:51.803119 6963 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:04:51.810988 12563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:04:51.811488 12564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:04:51.817214 12559 heartbeater.cc:347] Connected to a master server at 127.6.204.254:32811
I20240502 08:04:51.817642 12559 heartbeater.cc:464] Registering TS with master...
W20240502 08:04:51.818084 12566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:04:51.818554 6963 server_base.cc:1034] running on GCE node
I20240502 08:04:51.818682 12559 heartbeater.cc:511] Master 127.6.204.254:32811 requested a full tablet report, sending...
I20240502 08:04:51.819907 6963 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:04:51.820180 6963 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:04:51.820384 6963 hybrid_clock.cc:648] HybridClock initialized: now 1714637091820365 us; error 0 us; skew 500 ppm
I20240502 08:04:51.820979 6963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:04:51.821112 12432 ts_manager.cc:194] Registered new tserver with Master: 7e739706a3cb4269b73b3dc63e28ccfe (127.6.204.193:32777)
I20240502 08:04:51.823246 12432 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47736
I20240502 08:04:51.823803 6963 webserver.cc:460] Webserver started at http://127.6.204.194:40043/ using document root <none> and password file <none>
I20240502 08:04:51.824390 6963 fs_manager.cc:359] Metadata directory not provided
I20240502 08:04:51.824627 6963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:04:51.824970 6963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:04:51.826319 6963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714636933625983-6963-0/minicluster-data/ts-1-root/instance:
uuid: "ca3300cf09c04d6cac79551a5ab409ea"
format_stamp: "Formatted at 2024-05-02 08:04:51 on dist-test-slave-32xs"
I20240502 08:04:51.830765 6963 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240502 08:04:51.834013 12571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:04:51.834770 6963 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 08:04:51.835041 6963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714636933625983-6963-0/minicluster-data/ts-1-root
uuid: "ca3300cf09c04d6cac79551a5ab409ea"
format_stamp: "Formatted at 2024-05-02 08:04:51 on dist-test-slave-32xs"
I20240502 08:04:51.835279 6963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714636933625983-6963-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714636933625983-6963-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714636933625983-6963-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:04:51.851379 6963 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:04:51.852362 6963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:04:51.853749 6963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:04:51.856065 6963 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:04:51.856246 6963 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:04:51.856562 6963 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:04:51.856706 6963 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:04:51.893427 6963 rpc_server.cc:307] RPC server started. Bound to: 127.6.204.194:39897
I20240502 08:04:51.893546 12633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.204.194:39897 every 8 connection(s)
I20240502 08:04:51.898162 6963 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:04:51.904453 12637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:04:51.904716 12638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:04:51.908317 12640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:04:51.908442 6963 server_base.cc:1034] running on GCE node
I20240502 08:04:51.909561 6963 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20240502 08:04:51.909613 12634 heartbeater.cc:347] Connected to a master server at 127.6.204.254:32811
W20240502 08:04:51.909888 6963 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:04:51.910037 12634 heartbeater.cc:464] Registering TS with master...
I20240502 08:04:51.910070 6963 hybrid_clock.cc:648] HybridClock initialized: now 1714637091910055 us; error 0 us; skew 500 ppm
I20240502 08:04:51.910902 6963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:04:51.910984 12634 heartbeater.cc:511] Master 127.6.204.254:32811 requested a full tablet report, sending...
I20240502 08:04:51.913220 12432 ts_manager.cc:194] Registered new tserver with Master: ca3300cf09c04d6cac79551a5ab409ea (127.6.204.194:39897)
I20240502 08:04:51.913555 6963 webserver.cc:460] Webserver started at http://127.6.204.195:37607/ using document root <none> and password file <none>
I20240502 08:04:51.914213 6963 fs_manager.cc:359] Metadata directory not provided
I20240502 08:04:51.914448 6963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:04:51.914757 6963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:04:51.915083 12432 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47740
I20240502 08:04:51.916225 6963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714636933625983-6963-0/minicluster-data/ts-2-root/instance:
uuid: "b79fbd9f226f4270a50e10dce97bafad"
format_stamp: "Formatted at 2024-05-02 08:04:51 on dist-test-slave-32xs"
I20240502 08:04:51.920696 6963 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240502 08:04:51.923985 12645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:04:51.924823 6963 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 08:04:51.925127 6963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714636933625983-6963-0/minicluster-data/ts-2-root
uuid: "b79fbd9f226f4270a50e10dce97bafad"
format_stamp: "Formatted at 2024-05-02 08:04:51 on dist-test-slave-32xs"
I20240502 08:04:51.925395 6963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714636933625983-6963-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714636933625983-6963-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714636933625983-6963-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:04:51.937892 6963 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:04:51.938920 6963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:04:51.940302 6963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:04:51.942523 6963 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:04:51.942703 6963 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:04:51.942960 6963 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:04:51.943094 6963 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:04:51.979758 6963 rpc_server.cc:307] RPC server started. Bound to: 127.6.204.195:46407
I20240502 08:04:51.979848 12707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.204.195:46407 every 8 connection(s)
I20240502 08:04:51.992450 12708 heartbeater.cc:347] Connected to a master server at 127.6.204.254:32811
I20240502 08:04:51.992813 12708 heartbeater.cc:464] Registering TS with master...
I20240502 08:04:51.993474 12708 heartbeater.cc:511] Master 127.6.204.254:32811 requested a full tablet report, sending...
I20240502 08:04:51.995250 12432 ts_manager.cc:194] Registered new tserver with Master: b79fbd9f226f4270a50e10dce97bafad (127.6.204.195:46407)
I20240502 08:04:51.996156 6963 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013277354s
I20240502 08:04:51.996577 12432 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47742
I20240502 08:04:52.826247 12559 heartbeater.cc:503] Master 127.6.204.254:32811 was elected leader, sending a full tablet report...
I20240502 08:04:52.917623 12634 heartbeater.cc:503] Master 127.6.204.254:32811 was elected leader, sending a full tablet report...
I20240502 08:04:52.999020 12708 heartbeater.cc:503] Master 127.6.204.254:32811 was elected leader, sending a full tablet report...
I20240502 08:04:53.028602 6963 test_util.cc:268] Using random seed: 249119455
I20240502 08:04:53.055531 12432 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47756:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:04:53.057775 12432 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:04:53.117971 12673 tablet_service.cc:1467] Processing CreateTablet for tablet 1abb47c8c0a642b097ae6327e60ec0e0 (DEFAULT_TABLE table=test-workload [id=08210cb430b440d0a7d1151c17012950]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240502 08:04:53.118217 12672 tablet_service.cc:1467] Processing CreateTablet for tablet bde9453a83b8464899af354b5e24806f (DEFAULT_TABLE table=test-workload [id=08210cb430b440d0a7d1151c17012950]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240502 08:04:53.119563 12673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1abb47c8c0a642b097ae6327e60ec0e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:53.119959 12599 tablet_service.cc:1467] Processing CreateTablet for tablet 1abb47c8c0a642b097ae6327e60ec0e0 (DEFAULT_TABLE table=test-workload [id=08210cb430b440d0a7d1151c17012950]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240502 08:04:53.120056 12524 tablet_service.cc:1467] Processing CreateTablet for tablet 1abb47c8c0a642b097ae6327e60ec0e0 (DEFAULT_TABLE table=test-workload [id=08210cb430b440d0a7d1151c17012950]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240502 08:04:53.120991 12599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1abb47c8c0a642b097ae6327e60ec0e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:53.121307 12524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1abb47c8c0a642b097ae6327e60ec0e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:53.122956 12521 tablet_service.cc:1467] Processing CreateTablet for tablet 7fe7134eaf2e4e29ac762afe8855bdac (DEFAULT_TABLE table=test-workload [id=08210cb430b440d0a7d1151c17012950]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240502 08:04:53.123250 12523 tablet_service.cc:1467] Processing CreateTablet for tablet bde9453a83b8464899af354b5e24806f (DEFAULT_TABLE table=test-workload [id=08210cb430b440d0a7d1151c17012950]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240502 08:04:53.124233 12521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fe7134eaf2e4e29ac762afe8855bdac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:53.124866 12523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bde9453a83b8464899af354b5e24806f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:53.125319 12522 tablet_service.cc:1467] Processing CreateTablet for tablet 90b6d92c72f44d89a54c7b1b55470f4b (DEFAULT_TABLE table=test-workload [id=08210cb430b440d0a7d1151c17012950]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240502 08:04:53.126559 12522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90b6d92c72f44d89a54c7b1b55470f4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:53.126747 12672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bde9453a83b8464899af354b5e24806f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:53.126952 12598 tablet_service.cc:1467] Processing CreateTablet for tablet bde9453a83b8464899af354b5e24806f (DEFAULT_TABLE table=test-workload [id=08210cb430b440d0a7d1151c17012950]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240502 08:04:53.128248 12598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bde9453a83b8464899af354b5e24806f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:53.132367 12597 tablet_service.cc:1467] Processing CreateTablet for tablet 90b6d92c72f44d89a54c7b1b55470f4b (DEFAULT_TABLE table=test-workload [id=08210cb430b440d0a7d1151c17012950]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240502 08:04:53.133680 12597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90b6d92c72f44d89a54c7b1b55470f4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:53.137079 12596 tablet_service.cc:1467] Processing CreateTablet for tablet 7fe7134eaf2e4e29ac762afe8855bdac (DEFAULT_TABLE table=test-workload [id=08210cb430b440d0a7d1151c17012950]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240502 08:04:53.137913 12671 tablet_service.cc:1467] Processing CreateTablet for tablet 90b6d92c72f44d89a54c7b1b55470f4b (DEFAULT_TABLE table=test-workload [id=08210cb430b440d0a7d1151c17012950]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240502 08:04:53.138425 12596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fe7134eaf2e4e29ac762afe8855bdac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:53.139233 12671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90b6d92c72f44d89a54c7b1b55470f4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:53.155766 12670 tablet_service.cc:1467] Processing CreateTablet for tablet 7fe7134eaf2e4e29ac762afe8855bdac (DEFAULT_TABLE table=test-workload [id=08210cb430b440d0a7d1151c17012950]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240502 08:04:53.157136 12670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fe7134eaf2e4e29ac762afe8855bdac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:53.161209 12728 tablet_bootstrap.cc:492] T bde9453a83b8464899af354b5e24806f P 7e739706a3cb4269b73b3dc63e28ccfe: Bootstrap starting.
I20240502 08:04:53.166810 12728 tablet_bootstrap.cc:654] T bde9453a83b8464899af354b5e24806f P 7e739706a3cb4269b73b3dc63e28ccfe: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:53.178189 12728 tablet_bootstrap.cc:492] T bde9453a83b8464899af354b5e24806f P 7e739706a3cb4269b73b3dc63e28ccfe: No bootstrap required, opened a new log
I20240502 08:04:53.178761 12728 ts_tablet_manager.cc:1367] T bde9453a83b8464899af354b5e24806f P 7e739706a3cb4269b73b3dc63e28ccfe: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.009s
I20240502 08:04:53.181344 12728 raft_consensus.cc:348] T bde9453a83b8464899af354b5e24806f P 7e739706a3cb4269b73b3dc63e28ccfe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:53.182727 12728 raft_consensus.cc:374] T bde9453a83b8464899af354b5e24806f P 7e739706a3cb4269b73b3dc63e28ccfe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:53.183074 12728 raft_consensus.cc:729] T bde9453a83b8464899af354b5e24806f P 7e739706a3cb4269b73b3dc63e28ccfe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e739706a3cb4269b73b3dc63e28ccfe, State: Initialized, Role: FOLLOWER
I20240502 08:04:53.183897 12728 consensus_queue.cc:260] T bde9453a83b8464899af354b5e24806f P 7e739706a3cb4269b73b3dc63e28ccfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:53.186034 12730 tablet_bootstrap.cc:492] T bde9453a83b8464899af354b5e24806f P ca3300cf09c04d6cac79551a5ab409ea: Bootstrap starting.
I20240502 08:04:53.191320 12728 ts_tablet_manager.cc:1398] T bde9453a83b8464899af354b5e24806f P 7e739706a3cb4269b73b3dc63e28ccfe: Time spent starting tablet: real 0.012s user 0.002s sys 0.004s
I20240502 08:04:53.191576 12730 tablet_bootstrap.cc:654] T bde9453a83b8464899af354b5e24806f P ca3300cf09c04d6cac79551a5ab409ea: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:53.192399 12728 tablet_bootstrap.cc:492] T 1abb47c8c0a642b097ae6327e60ec0e0 P 7e739706a3cb4269b73b3dc63e28ccfe: Bootstrap starting.
I20240502 08:04:53.198208 12728 tablet_bootstrap.cc:654] T 1abb47c8c0a642b097ae6327e60ec0e0 P 7e739706a3cb4269b73b3dc63e28ccfe: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:53.210860 12728 tablet_bootstrap.cc:492] T 1abb47c8c0a642b097ae6327e60ec0e0 P 7e739706a3cb4269b73b3dc63e28ccfe: No bootstrap required, opened a new log
I20240502 08:04:53.211485 12728 ts_tablet_manager.cc:1367] T 1abb47c8c0a642b097ae6327e60ec0e0 P 7e739706a3cb4269b73b3dc63e28ccfe: Time spent bootstrapping tablet: real 0.019s user 0.014s sys 0.002s
I20240502 08:04:53.214100 12728 raft_consensus.cc:348] T 1abb47c8c0a642b097ae6327e60ec0e0 P 7e739706a3cb4269b73b3dc63e28ccfe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:53.218533 12728 raft_consensus.cc:374] T 1abb47c8c0a642b097ae6327e60ec0e0 P 7e739706a3cb4269b73b3dc63e28ccfe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:53.218888 12728 raft_consensus.cc:729] T 1abb47c8c0a642b097ae6327e60ec0e0 P 7e739706a3cb4269b73b3dc63e28ccfe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e739706a3cb4269b73b3dc63e28ccfe, State: Initialized, Role: FOLLOWER
I20240502 08:04:53.220299 12734 tablet_bootstrap.cc:492] T 7fe7134eaf2e4e29ac762afe8855bdac P b79fbd9f226f4270a50e10dce97bafad: Bootstrap starting.
I20240502 08:04:53.219856 12728 consensus_queue.cc:260] T 1abb47c8c0a642b097ae6327e60ec0e0 P 7e739706a3cb4269b73b3dc63e28ccfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:53.221563 12730 tablet_bootstrap.cc:492] T bde9453a83b8464899af354b5e24806f P ca3300cf09c04d6cac79551a5ab409ea: No bootstrap required, opened a new log
I20240502 08:04:53.222103 12730 ts_tablet_manager.cc:1367] T bde9453a83b8464899af354b5e24806f P ca3300cf09c04d6cac79551a5ab409ea: Time spent bootstrapping tablet: real 0.036s user 0.018s sys 0.007s
I20240502 08:04:53.222168 12728 ts_tablet_manager.cc:1398] T 1abb47c8c0a642b097ae6327e60ec0e0 P 7e739706a3cb4269b73b3dc63e28ccfe: Time spent starting tablet: real 0.010s user 0.001s sys 0.004s
I20240502 08:04:53.223317 12728 tablet_bootstrap.cc:492] T 90b6d92c72f44d89a54c7b1b55470f4b P 7e739706a3cb4269b73b3dc63e28ccfe: Bootstrap starting.
I20240502 08:04:53.224530 12730 raft_consensus.cc:348] T bde9453a83b8464899af354b5e24806f P ca3300cf09c04d6cac79551a5ab409ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:53.225244 12730 raft_consensus.cc:374] T bde9453a83b8464899af354b5e24806f P ca3300cf09c04d6cac79551a5ab409ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:53.225528 12730 raft_consensus.cc:729] T bde9453a83b8464899af354b5e24806f P ca3300cf09c04d6cac79551a5ab409ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca3300cf09c04d6cac79551a5ab409ea, State: Initialized, Role: FOLLOWER
I20240502 08:04:53.226357 12730 consensus_queue.cc:260] T bde9453a83b8464899af354b5e24806f P ca3300cf09c04d6cac79551a5ab409ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:53.229290 12728 tablet_bootstrap.cc:654] T 90b6d92c72f44d89a54c7b1b55470f4b P 7e739706a3cb4269b73b3dc63e28ccfe: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:53.229743 12734 tablet_bootstrap.cc:654] T 7fe7134eaf2e4e29ac762afe8855bdac P b79fbd9f226f4270a50e10dce97bafad: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:53.237028 12730 ts_tablet_manager.cc:1398] T bde9453a83b8464899af354b5e24806f P ca3300cf09c04d6cac79551a5ab409ea: Time spent starting tablet: real 0.014s user 0.007s sys 0.006s
I20240502 08:04:53.238420 12730 tablet_bootstrap.cc:492] T 90b6d92c72f44d89a54c7b1b55470f4b P ca3300cf09c04d6cac79551a5ab409ea: Bootstrap starting.
I20240502 08:04:53.238301 12728 tablet_bootstrap.cc:492] T 90b6d92c72f44d89a54c7b1b55470f4b P 7e739706a3cb4269b73b3dc63e28ccfe: No bootstrap required, opened a new log
I20240502 08:04:53.239920 12728 ts_tablet_manager.cc:1367] T 90b6d92c72f44d89a54c7b1b55470f4b P 7e739706a3cb4269b73b3dc63e28ccfe: Time spent bootstrapping tablet: real 0.017s user 0.015s sys 0.000s
I20240502 08:04:53.239120 12734 tablet_bootstrap.cc:492] T 7fe7134eaf2e4e29ac762afe8855bdac P b79fbd9f226f4270a50e10dce97bafad: No bootstrap required, opened a new log
I20240502 08:04:53.243028 12734 ts_tablet_manager.cc:1367] T 7fe7134eaf2e4e29ac762afe8855bdac P b79fbd9f226f4270a50e10dce97bafad: Time spent bootstrapping tablet: real 0.023s user 0.006s sys 0.010s
I20240502 08:04:53.242676 12728 raft_consensus.cc:348] T 90b6d92c72f44d89a54c7b1b55470f4b P 7e739706a3cb4269b73b3dc63e28ccfe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:53.243520 12728 raft_consensus.cc:374] T 90b6d92c72f44d89a54c7b1b55470f4b P 7e739706a3cb4269b73b3dc63e28ccfe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:53.243894 12728 raft_consensus.cc:729] T 90b6d92c72f44d89a54c7b1b55470f4b P 7e739706a3cb4269b73b3dc63e28ccfe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e739706a3cb4269b73b3dc63e28ccfe, State: Initialized, Role: FOLLOWER
I20240502 08:04:53.244115 12730 tablet_bootstrap.cc:654] T 90b6d92c72f44d89a54c7b1b55470f4b P ca3300cf09c04d6cac79551a5ab409ea: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:53.244762 12728 consensus_queue.cc:260] T 90b6d92c72f44d89a54c7b1b55470f4b P 7e739706a3cb4269b73b3dc63e28ccfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:53.245703 12734 raft_consensus.cc:348] T 7fe7134eaf2e4e29ac762afe8855bdac P b79fbd9f226f4270a50e10dce97bafad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:53.246575 12734 raft_consensus.cc:374] T 7fe7134eaf2e4e29ac762afe8855bdac P b79fbd9f226f4270a50e10dce97bafad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:53.246909 12734 raft_consensus.cc:729] T 7fe7134eaf2e4e29ac762afe8855bdac P b79fbd9f226f4270a50e10dce97bafad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b79fbd9f226f4270a50e10dce97bafad, State: Initialized, Role: FOLLOWER
I20240502 08:04:53.247294 12728 ts_tablet_manager.cc:1398] T 90b6d92c72f44d89a54c7b1b55470f4b P 7e739706a3cb4269b73b3dc63e28ccfe: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240502 08:04:53.247887 12734 consensus_queue.cc:260] T 7fe7134eaf2e4e29ac762afe8855bdac P b79fbd9f226f4270a50e10dce97bafad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:53.248533 12728 tablet_bootstrap.cc:492] T 7fe7134eaf2e4e29ac762afe8855bdac P 7e739706a3cb4269b73b3dc63e28ccfe: Bootstrap starting.
I20240502 08:04:53.254725 12728 tablet_bootstrap.cc:654] T 7fe7134eaf2e4e29ac762afe8855bdac P 7e739706a3cb4269b73b3dc63e28ccfe: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:53.255272 12730 tablet_bootstrap.cc:492] T 90b6d92c72f44d89a54c7b1b55470f4b P ca3300cf09c04d6cac79551a5ab409ea: No bootstrap required, opened a new log
I20240502 08:04:53.255790 12730 ts_tablet_manager.cc:1367] T 90b6d92c72f44d89a54c7b1b55470f4b P ca3300cf09c04d6cac79551a5ab409ea: Time spent bootstrapping tablet: real 0.018s user 0.016s sys 0.000s
I20240502 08:04:53.258350 12730 raft_consensus.cc:348] T 90b6d92c72f44d89a54c7b1b55470f4b P ca3300cf09c04d6cac79551a5ab409ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:53.259124 12730 raft_consensus.cc:374] T 90b6d92c72f44d89a54c7b1b55470f4b P ca3300cf09c04d6cac79551a5ab409ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:53.259461 12730 raft_consensus.cc:729] T 90b6d92c72f44d89a54c7b1b55470f4b P ca3300cf09c04d6cac79551a5ab409ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca3300cf09c04d6cac79551a5ab409ea, State: Initialized, Role: FOLLOWER
I20240502 08:04:53.260135 12730 consensus_queue.cc:260] T 90b6d92c72f44d89a54c7b1b55470f4b P ca3300cf09c04d6cac79551a5ab409ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:53.261960 12730 ts_tablet_manager.cc:1398] T 90b6d92c72f44d89a54c7b1b55470f4b P ca3300cf09c04d6cac79551a5ab409ea: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240502 08:04:53.262820 12734 ts_tablet_manager.cc:1398] T 7fe7134eaf2e4e29ac762afe8855bdac P b79fbd9f226f4270a50e10dce97bafad: Time spent starting tablet: real 0.019s user 0.009s sys 0.004s
I20240502 08:04:53.263005 12730 tablet_bootstrap.cc:492] T 1abb47c8c0a642b097ae6327e60ec0e0 P ca3300cf09c04d6cac79551a5ab409ea: Bootstrap starting.
I20240502 08:04:53.266495 12728 tablet_bootstrap.cc:492] T 7fe7134eaf2e4e29ac762afe8855bdac P 7e739706a3cb4269b73b3dc63e28ccfe: No bootstrap required, opened a new log
I20240502 08:04:53.267060 12728 ts_tablet_manager.cc:1367] T 7fe7134eaf2e4e29ac762afe8855bdac P 7e739706a3cb4269b73b3dc63e28ccfe: Time spent bootstrapping tablet: real 0.019s user 0.012s sys 0.005s
I20240502 08:04:53.268918 12730 tablet_bootstrap.cc:654] T 1abb47c8c0a642b097ae6327e60ec0e0 P ca3300cf09c04d6cac79551a5ab409ea: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:53.270023 12734 tablet_bootstrap.cc:492] T bde9453a83b8464899af354b5e24806f P b79fbd9f226f4270a50e10dce97bafad: Bootstrap starting.
I20240502 08:04:53.269557 12728 raft_consensus.cc:348] T 7fe7134eaf2e4e29ac762afe8855bdac P 7e739706a3cb4269b73b3dc63e28ccfe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:53.270486 12728 raft_consensus.cc:374] T 7fe7134eaf2e4e29ac762afe8855bdac P 7e739706a3cb4269b73b3dc63e28ccfe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:53.270818 12728 raft_consensus.cc:729] T 7fe7134eaf2e4e29ac762afe8855bdac P 7e739706a3cb4269b73b3dc63e28ccfe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e739706a3cb4269b73b3dc63e28ccfe, State: Initialized, Role: FOLLOWER
I20240502 08:04:53.271612 12728 consensus_queue.cc:260] T 7fe7134eaf2e4e29ac762afe8855bdac P 7e739706a3cb4269b73b3dc63e28ccfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:53.275612 12728 ts_tablet_manager.cc:1398] T 7fe7134eaf2e4e29ac762afe8855bdac P 7e739706a3cb4269b73b3dc63e28ccfe: Time spent starting tablet: real 0.008s user 0.004s sys 0.001s
I20240502 08:04:53.280053 12734 tablet_bootstrap.cc:654] T bde9453a83b8464899af354b5e24806f P b79fbd9f226f4270a50e10dce97bafad: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:53.285501 12734 tablet_bootstrap.cc:492] T bde9453a83b8464899af354b5e24806f P b79fbd9f226f4270a50e10dce97bafad: No bootstrap required, opened a new log
I20240502 08:04:53.285996 12734 ts_tablet_manager.cc:1367] T bde9453a83b8464899af354b5e24806f P b79fbd9f226f4270a50e10dce97bafad: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.002s
I20240502 08:04:53.286396 12730 tablet_bootstrap.cc:492] T 1abb47c8c0a642b097ae6327e60ec0e0 P ca3300cf09c04d6cac79551a5ab409ea: No bootstrap required, opened a new log
I20240502 08:04:53.286877 12730 ts_tablet_manager.cc:1367] T 1abb47c8c0a642b097ae6327e60ec0e0 P ca3300cf09c04d6cac79551a5ab409ea: Time spent bootstrapping tablet: real 0.024s user 0.011s sys 0.011s
I20240502 08:04:53.288553 12734 raft_consensus.cc:348] T bde9453a83b8464899af354b5e24806f P b79fbd9f226f4270a50e10dce97bafad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:53.289331 12734 raft_consensus.cc:374] T bde9453a83b8464899af354b5e24806f P b79fbd9f226f4270a50e10dce97bafad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:53.289767 12734 raft_consensus.cc:729] T bde9453a83b8464899af354b5e24806f P b79fbd9f226f4270a50e10dce97bafad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b79fbd9f226f4270a50e10dce97bafad, State: Initialized, Role: FOLLOWER
I20240502 08:04:53.289351 12730 raft_consensus.cc:348] T 1abb47c8c0a642b097ae6327e60ec0e0 P ca3300cf09c04d6cac79551a5ab409ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:53.290251 12730 raft_consensus.cc:374] T 1abb47c8c0a642b097ae6327e60ec0e0 P ca3300cf09c04d6cac79551a5ab409ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:53.290540 12730 raft_consensus.cc:729] T 1abb47c8c0a642b097ae6327e60ec0e0 P ca3300cf09c04d6cac79551a5ab409ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca3300cf09c04d6cac79551a5ab409ea, State: Initialized, Role: FOLLOWER
I20240502 08:04:53.290532 12734 consensus_queue.cc:260] T bde9453a83b8464899af354b5e24806f P b79fbd9f226f4270a50e10dce97bafad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:53.291432 12730 consensus_queue.cc:260] T 1abb47c8c0a642b097ae6327e60ec0e0 P ca3300cf09c04d6cac79551a5ab409ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:53.292913 12734 ts_tablet_manager.cc:1398] T bde9453a83b8464899af354b5e24806f P b79fbd9f226f4270a50e10dce97bafad: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240502 08:04:53.293579 12730 ts_tablet_manager.cc:1398] T 1abb47c8c0a642b097ae6327e60ec0e0 P ca3300cf09c04d6cac79551a5ab409ea: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240502 08:04:53.293881 12734 tablet_bootstrap.cc:492] T 90b6d92c72f44d89a54c7b1b55470f4b P b79fbd9f226f4270a50e10dce97bafad: Bootstrap starting.
I20240502 08:04:53.294584 12730 tablet_bootstrap.cc:492] T 7fe7134eaf2e4e29ac762afe8855bdac P ca3300cf09c04d6cac79551a5ab409ea: Bootstrap starting.
I20240502 08:04:53.300168 12730 tablet_bootstrap.cc:654] T 7fe7134eaf2e4e29ac762afe8855bdac P ca3300cf09c04d6cac79551a5ab409ea: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:53.300168 12734 tablet_bootstrap.cc:654] T 90b6d92c72f44d89a54c7b1b55470f4b P b79fbd9f226f4270a50e10dce97bafad: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:53.305761 12730 tablet_bootstrap.cc:492] T 7fe7134eaf2e4e29ac762afe8855bdac P ca3300cf09c04d6cac79551a5ab409ea: No bootstrap required, opened a new log
I20240502 08:04:53.306213 12730 ts_tablet_manager.cc:1367] T 7fe7134eaf2e4e29ac762afe8855bdac P ca3300cf09c04d6cac79551a5ab409ea: Time spent bootstrapping tablet: real 0.012s user 0.011s sys 0.000s
I20240502 08:04:53.308116 12734 tablet_bootstrap.cc:492] T 90b6d92c72f44d89a54c7b1b55470f4b P b79fbd9f226f4270a50e10dce97bafad: No bootstrap required, opened a new log
I20240502 08:04:53.308540 12739 raft_consensus.cc:482] T 7fe7134eaf2e4e29ac762afe8855bdac P b79fbd9f226f4270a50e10dce97bafad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:04:53.308650 12734 ts_tablet_manager.cc:1367] T 90b6d92c72f44d89a54c7b1b55470f4b P b79fbd9f226f4270a50e10dce97bafad: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.005s
I20240502 08:04:53.308809 12730 raft_consensus.cc:348] T 7fe7134eaf2e4e29ac762afe8855bdac P ca3300cf09c04d6cac79551a5ab409ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:53.308977 12739 raft_consensus.cc:504] T 7fe7134eaf2e4e29ac762afe8855bdac P b79fbd9f226f4270a50e10dce97bafad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:53.309630 12730 raft_consensus.cc:374] T 7fe7134eaf2e4e29ac762afe8855bdac P ca3300cf09c04d6cac79551a5ab409ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:53.309942 12730 raft_consensus.cc:729] T 7fe7134eaf2e4e29ac762afe8855bdac P ca3300cf09c04d6cac79551a5ab409ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca3300cf09c04d6cac79551a5ab409ea, State: Initialized, Role: FOLLOWER
I20240502 08:04:53.310678 12730 consensus_queue.cc:260] T 7fe7134eaf2e4e29ac762afe8855bdac P ca3300cf09c04d6cac79551a5ab409ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:53.311432 12739 leader_election.cc:288] T 7fe7134eaf2e4e29ac762afe8855bdac P b79fbd9f226f4270a50e10dce97bafad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e739706a3cb4269b73b3dc63e28ccfe (127.6.204.193:32777), ca3300cf09c04d6cac79551a5ab409ea (127.6.204.194:39897)
I20240502 08:04:53.311883 12734 raft_consensus.cc:348] T 90b6d92c72f44d89a54c7b1b55470f4b P b79fbd9f226f4270a50e10dce97bafad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:53.312877 12730 ts_tablet_manager.cc:1398] T 7fe7134eaf2e4e29ac762afe8855bdac P ca3300cf09c04d6cac79551a5ab409ea: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240502 08:04:53.312618 12734 raft_consensus.cc:374] T 90b6d92c72f44d89a54c7b1b55470f4b P b79fbd9f226f4270a50e10dce97bafad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:53.313659 12734 raft_consensus.cc:729] T 90b6d92c72f44d89a54c7b1b55470f4b P b79fbd9f226f4270a50e10dce97bafad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b79fbd9f226f4270a50e10dce97bafad, State: Initialized, Role: FOLLOWER
I20240502 08:04:53.314287 12734 consensus_queue.cc:260] T 90b6d92c72f44d89a54c7b1b55470f4b P b79fbd9f226f4270a50e10dce97bafad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:53.316278 12734 ts_tablet_manager.cc:1398] T 90b6d92c72f44d89a54c7b1b55470f4b P b79fbd9f226f4270a50e10dce97bafad: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240502 08:04:53.317299 12734 tablet_bootstrap.cc:492] T 1abb47c8c0a642b097ae6327e60ec0e0 P b79fbd9f226f4270a50e10dce97bafad: Bootstrap starting.
I20240502 08:04:53.323222 12734 tablet_bootstrap.cc:654] T 1abb47c8c0a642b097ae6327e60ec0e0 P b79fbd9f226f4270a50e10dce97bafad: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:53.334834 12735 raft_consensus.cc:482] T 7fe7134eaf2e4e29ac762afe8855bdac P ca3300cf09c04d6cac79551a5ab409ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:04:53.335719 12734 tablet_bootstrap.cc:492] T 1abb47c8c0a642b097ae6327e60ec0e0 P b79fbd9f226f4270a50e10dce97bafad: No bootstrap required, opened a new log
I20240502 08:04:53.335356 12534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fe7134eaf2e4e29ac762afe8855bdac" candidate_uuid: "b79fbd9f226f4270a50e10dce97bafad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" is_pre_election: true
I20240502 08:04:53.335426 12735 raft_consensus.cc:504] T 7fe7134eaf2e4e29ac762afe8855bdac P ca3300cf09c04d6cac79551a5ab409ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:53.336287 12734 ts_tablet_manager.cc:1367] T 1abb47c8c0a642b097ae6327e60ec0e0 P b79fbd9f226f4270a50e10dce97bafad: Time spent bootstrapping tablet: real 0.019s user 0.012s sys 0.004s
I20240502 08:04:53.336436 12534 raft_consensus.cc:2454] T 7fe7134eaf2e4e29ac762afe8855bdac P 7e739706a3cb4269b73b3dc63e28ccfe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b79fbd9f226f4270a50e10dce97bafad in term 0.
I20240502 08:04:53.337817 12647 leader_election.cc:302] T 7fe7134eaf2e4e29ac762afe8855bdac P b79fbd9f226f4270a50e10dce97bafad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e739706a3cb4269b73b3dc63e28ccfe, b79fbd9f226f4270a50e10dce97bafad; no voters:
I20240502 08:04:53.338661 12739 raft_consensus.cc:2781] T 7fe7134eaf2e4e29ac762afe8855bdac P b79fbd9f226f4270a50e10dce97bafad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:04:53.339037 12739 raft_consensus.cc:482] T 7fe7134eaf2e4e29ac762afe8855bdac P b79fbd9f226f4270a50e10dce97bafad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:04:53.338961 12734 raft_consensus.cc:348] T 1abb47c8c0a642b097ae6327e60ec0e0 P b79fbd9f226f4270a50e10dce97bafad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:53.339913 12734 raft_consensus.cc:374] T 1abb47c8c0a642b097ae6327e60ec0e0 P b79fbd9f226f4270a50e10dce97bafad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:53.339593 12739 raft_consensus.cc:3037] T 7fe7134eaf2e4e29ac762afe8855bdac P b79fbd9f226f4270a50e10dce97bafad [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:53.340232 12734 raft_consensus.cc:729] T 1abb47c8c0a642b097ae6327e60ec0e0 P b79fbd9f226f4270a50e10dce97bafad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b79fbd9f226f4270a50e10dce97bafad, State: Initialized, Role: FOLLOWER
I20240502 08:04:53.341053 12734 consensus_queue.cc:260] T 1abb47c8c0a642b097ae6327e60ec0e0 P b79fbd9f226f4270a50e10dce97bafad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:53.345716 12735 leader_election.cc:288] T 7fe7134eaf2e4e29ac762afe8855bdac P ca3300cf09c04d6cac79551a5ab409ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e739706a3cb4269b73b3dc63e28ccfe (127.6.204.193:32777), b79fbd9f226f4270a50e10dce97bafad (127.6.204.195:46407)
I20240502 08:04:53.349957 12739 raft_consensus.cc:504] T 7fe7134eaf2e4e29ac762afe8855bdac P b79fbd9f226f4270a50e10dce97bafad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:53.352965 12534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fe7134eaf2e4e29ac762afe8855bdac" candidate_uuid: "b79fbd9f226f4270a50e10dce97bafad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e739706a3cb4269b73b3dc63e28ccfe"
I20240502 08:04:53.353582 12534 raft_consensus.cc:3037] T 7fe7134eaf2e4e29ac762afe8855bdac P 7e739706a3cb4269b73b3dc63e28ccfe [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:53.358278 12609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fe7134eaf2e4e29ac762afe8855bdac" candidate_uuid: "b79fbd9f226f4270a50e10dce97bafad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca3300cf09c04d6cac79551a5ab409ea" is_pre_election: true
I20240502 08:04:53.358773 12739 leader_election.cc:288] T 7fe7134eaf2e4e29ac762afe8855bdac P b79fbd9f226f4270a50e10dce97bafad [CANDIDATE]: Term 1 election: Requested vote from peers 7e739706a3cb4269b73b3dc63e28ccfe (127.6.204.193:32777), ca3300cf09c04d6cac79551a5ab409ea (127.6.204.194:39897)
I20240502 08:04:53.359342 12609 raft_consensus.cc:2454] T 7fe7134eaf2e4e29ac762afe8855bdac P ca3300cf09c04d6cac79551a5ab409ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b79fbd9f226f4270a50e10dce97bafad in term 0.
I20240502 08:04:53.361923 12609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fe7134eaf2e4e29ac762afe8855bdac" candidate_uuid: "b79fbd9f226f4270a50e10dce97bafad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca3300cf09c04d6cac79551a5ab409ea"
I20240502 08:04:53.362697 12609 raft_consensus.cc:3037] T 7fe7134eaf2e4e29ac762afe8855bdac P ca3300cf09c04d6cac79551a5ab409ea [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:53.363889 12533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fe7134eaf2e4e29ac762afe8855bdac" candidate_uuid: "ca3300cf09c04d6cac79551a5ab409ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" is_pre_election: true
I20240502 08:04:53.362673 12534 raft_consensus.cc:2454] T 7fe7134eaf2e4e29ac762afe8855bdac P 7e739706a3cb4269b73b3dc63e28ccfe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b79fbd9f226f4270a50e10dce97bafad in term 1.
I20240502 08:04:53.366923 12647 leader_election.cc:302] T 7fe7134eaf2e4e29ac762afe8855bdac P b79fbd9f226f4270a50e10dce97bafad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e739706a3cb4269b73b3dc63e28ccfe, b79fbd9f226f4270a50e10dce97bafad; no voters:
I20240502 08:04:53.367977 12739 raft_consensus.cc:2781] T 7fe7134eaf2e4e29ac762afe8855bdac P b79fbd9f226f4270a50e10dce97bafad [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:04:53.368593 12739 raft_consensus.cc:686] T 7fe7134eaf2e4e29ac762afe8855bdac P b79fbd9f226f4270a50e10dce97bafad [term 1 LEADER]: Becoming Leader. State: Replica: b79fbd9f226f4270a50e10dce97bafad, State: Running, Role: LEADER
I20240502 08:04:53.369629 12739 consensus_queue.cc:237] T 7fe7134eaf2e4e29ac762afe8855bdac P b79fbd9f226f4270a50e10dce97bafad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:53.371377 12609 raft_consensus.cc:2454] T 7fe7134eaf2e4e29ac762afe8855bdac P ca3300cf09c04d6cac79551a5ab409ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b79fbd9f226f4270a50e10dce97bafad in term 1.
I20240502 08:04:53.373688 12734 ts_tablet_manager.cc:1398] T 1abb47c8c0a642b097ae6327e60ec0e0 P b79fbd9f226f4270a50e10dce97bafad: Time spent starting tablet: real 0.037s user 0.011s sys 0.027s
I20240502 08:04:53.378696 12430 catalog_manager.cc:5484] T 7fe7134eaf2e4e29ac762afe8855bdac P b79fbd9f226f4270a50e10dce97bafad reported cstate change: term changed from 0 to 1, leader changed from <none> to b79fbd9f226f4270a50e10dce97bafad (127.6.204.195). New cstate: current_term: 1 leader_uuid: "b79fbd9f226f4270a50e10dce97bafad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } health_report { overall_health: HEALTHY } } }
I20240502 08:04:53.380877 12683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fe7134eaf2e4e29ac762afe8855bdac" candidate_uuid: "ca3300cf09c04d6cac79551a5ab409ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b79fbd9f226f4270a50e10dce97bafad" is_pre_election: true
I20240502 08:04:53.382339 12574 leader_election.cc:302] T 7fe7134eaf2e4e29ac762afe8855bdac P ca3300cf09c04d6cac79551a5ab409ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca3300cf09c04d6cac79551a5ab409ea; no voters: 7e739706a3cb4269b73b3dc63e28ccfe, b79fbd9f226f4270a50e10dce97bafad
I20240502 08:04:53.383132 12735 raft_consensus.cc:2726] T 7fe7134eaf2e4e29ac762afe8855bdac P ca3300cf09c04d6cac79551a5ab409ea [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:04:53.386425 12739 raft_consensus.cc:482] T bde9453a83b8464899af354b5e24806f P b79fbd9f226f4270a50e10dce97bafad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:04:53.386901 12739 raft_consensus.cc:504] T bde9453a83b8464899af354b5e24806f P b79fbd9f226f4270a50e10dce97bafad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:53.388736 12739 leader_election.cc:288] T bde9453a83b8464899af354b5e24806f P b79fbd9f226f4270a50e10dce97bafad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e739706a3cb4269b73b3dc63e28ccfe (127.6.204.193:32777), ca3300cf09c04d6cac79551a5ab409ea (127.6.204.194:39897)
I20240502 08:04:53.389467 12534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bde9453a83b8464899af354b5e24806f" candidate_uuid: "b79fbd9f226f4270a50e10dce97bafad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" is_pre_election: true
I20240502 08:04:53.389704 12609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bde9453a83b8464899af354b5e24806f" candidate_uuid: "b79fbd9f226f4270a50e10dce97bafad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca3300cf09c04d6cac79551a5ab409ea" is_pre_election: true
I20240502 08:04:53.390148 12534 raft_consensus.cc:2454] T bde9453a83b8464899af354b5e24806f P 7e739706a3cb4269b73b3dc63e28ccfe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b79fbd9f226f4270a50e10dce97bafad in term 0.
I20240502 08:04:53.390306 12609 raft_consensus.cc:2454] T bde9453a83b8464899af354b5e24806f P ca3300cf09c04d6cac79551a5ab409ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b79fbd9f226f4270a50e10dce97bafad in term 0.
I20240502 08:04:53.391161 12647 leader_election.cc:302] T bde9453a83b8464899af354b5e24806f P b79fbd9f226f4270a50e10dce97bafad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e739706a3cb4269b73b3dc63e28ccfe, b79fbd9f226f4270a50e10dce97bafad; no voters:
I20240502 08:04:53.391793 12739 raft_consensus.cc:2781] T bde9453a83b8464899af354b5e24806f P b79fbd9f226f4270a50e10dce97bafad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:04:53.392164 12739 raft_consensus.cc:482] T bde9453a83b8464899af354b5e24806f P b79fbd9f226f4270a50e10dce97bafad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:04:53.392489 12739 raft_consensus.cc:3037] T bde9453a83b8464899af354b5e24806f P b79fbd9f226f4270a50e10dce97bafad [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:53.397202 12739 raft_consensus.cc:504] T bde9453a83b8464899af354b5e24806f P b79fbd9f226f4270a50e10dce97bafad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:53.398654 12739 leader_election.cc:288] T bde9453a83b8464899af354b5e24806f P b79fbd9f226f4270a50e10dce97bafad [CANDIDATE]: Term 1 election: Requested vote from peers 7e739706a3cb4269b73b3dc63e28ccfe (127.6.204.193:32777), ca3300cf09c04d6cac79551a5ab409ea (127.6.204.194:39897)
I20240502 08:04:53.399423 12534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bde9453a83b8464899af354b5e24806f" candidate_uuid: "b79fbd9f226f4270a50e10dce97bafad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e739706a3cb4269b73b3dc63e28ccfe"
I20240502 08:04:53.399930 12534 raft_consensus.cc:3037] T bde9453a83b8464899af354b5e24806f P 7e739706a3cb4269b73b3dc63e28ccfe [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:53.399679 12609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bde9453a83b8464899af354b5e24806f" candidate_uuid: "b79fbd9f226f4270a50e10dce97bafad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca3300cf09c04d6cac79551a5ab409ea"
I20240502 08:04:53.400409 12609 raft_consensus.cc:3037] T bde9453a83b8464899af354b5e24806f P ca3300cf09c04d6cac79551a5ab409ea [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:53.404508 12731 raft_consensus.cc:482] T 1abb47c8c0a642b097ae6327e60ec0e0 P 7e739706a3cb4269b73b3dc63e28ccfe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:04:53.404551 12534 raft_consensus.cc:2454] T bde9453a83b8464899af354b5e24806f P 7e739706a3cb4269b73b3dc63e28ccfe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b79fbd9f226f4270a50e10dce97bafad in term 1.
I20240502 08:04:53.404902 12609 raft_consensus.cc:2454] T bde9453a83b8464899af354b5e24806f P ca3300cf09c04d6cac79551a5ab409ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b79fbd9f226f4270a50e10dce97bafad in term 1.
I20240502 08:04:53.405133 12731 raft_consensus.cc:504] T 1abb47c8c0a642b097ae6327e60ec0e0 P 7e739706a3cb4269b73b3dc63e28ccfe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:53.405686 12647 leader_election.cc:302] T bde9453a83b8464899af354b5e24806f P b79fbd9f226f4270a50e10dce97bafad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e739706a3cb4269b73b3dc63e28ccfe, b79fbd9f226f4270a50e10dce97bafad; no voters:
I20240502 08:04:53.406414 12739 raft_consensus.cc:2781] T bde9453a83b8464899af354b5e24806f P b79fbd9f226f4270a50e10dce97bafad [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:04:53.406879 12739 raft_consensus.cc:686] T bde9453a83b8464899af354b5e24806f P b79fbd9f226f4270a50e10dce97bafad [term 1 LEADER]: Becoming Leader. State: Replica: b79fbd9f226f4270a50e10dce97bafad, State: Running, Role: LEADER
I20240502 08:04:53.407528 12731 leader_election.cc:288] T 1abb47c8c0a642b097ae6327e60ec0e0 P 7e739706a3cb4269b73b3dc63e28ccfe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ca3300cf09c04d6cac79551a5ab409ea (127.6.204.194:39897), b79fbd9f226f4270a50e10dce97bafad (127.6.204.195:46407)
I20240502 08:04:53.407625 12739 consensus_queue.cc:237] T bde9453a83b8464899af354b5e24806f P b79fbd9f226f4270a50e10dce97bafad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:53.417194 12430 catalog_manager.cc:5484] T bde9453a83b8464899af354b5e24806f P b79fbd9f226f4270a50e10dce97bafad reported cstate change: term changed from 0 to 1, leader changed from <none> to b79fbd9f226f4270a50e10dce97bafad (127.6.204.195). New cstate: current_term: 1 leader_uuid: "b79fbd9f226f4270a50e10dce97bafad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } health_report { overall_health: HEALTHY } } }
I20240502 08:04:53.420531 12683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1abb47c8c0a642b097ae6327e60ec0e0" candidate_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b79fbd9f226f4270a50e10dce97bafad" is_pre_election: true
I20240502 08:04:53.421264 12683 raft_consensus.cc:2454] T 1abb47c8c0a642b097ae6327e60ec0e0 P b79fbd9f226f4270a50e10dce97bafad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e739706a3cb4269b73b3dc63e28ccfe in term 0.
I20240502 08:04:53.422328 12499 leader_election.cc:302] T 1abb47c8c0a642b097ae6327e60ec0e0 P 7e739706a3cb4269b73b3dc63e28ccfe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e739706a3cb4269b73b3dc63e28ccfe, b79fbd9f226f4270a50e10dce97bafad; no voters:
I20240502 08:04:53.423178 12731 raft_consensus.cc:2781] T 1abb47c8c0a642b097ae6327e60ec0e0 P 7e739706a3cb4269b73b3dc63e28ccfe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:04:53.423528 12731 raft_consensus.cc:482] T 1abb47c8c0a642b097ae6327e60ec0e0 P 7e739706a3cb4269b73b3dc63e28ccfe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:04:53.423429 12609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1abb47c8c0a642b097ae6327e60ec0e0" candidate_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca3300cf09c04d6cac79551a5ab409ea" is_pre_election: true
I20240502 08:04:53.423990 12731 raft_consensus.cc:3037] T 1abb47c8c0a642b097ae6327e60ec0e0 P 7e739706a3cb4269b73b3dc63e28ccfe [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:53.424197 12609 raft_consensus.cc:2454] T 1abb47c8c0a642b097ae6327e60ec0e0 P ca3300cf09c04d6cac79551a5ab409ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e739706a3cb4269b73b3dc63e28ccfe in term 0.
I20240502 08:04:53.430660 12731 raft_consensus.cc:504] T 1abb47c8c0a642b097ae6327e60ec0e0 P 7e739706a3cb4269b73b3dc63e28ccfe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:53.432252 12731 leader_election.cc:288] T 1abb47c8c0a642b097ae6327e60ec0e0 P 7e739706a3cb4269b73b3dc63e28ccfe [CANDIDATE]: Term 1 election: Requested vote from peers ca3300cf09c04d6cac79551a5ab409ea (127.6.204.194:39897), b79fbd9f226f4270a50e10dce97bafad (127.6.204.195:46407)
I20240502 08:04:53.432948 12609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1abb47c8c0a642b097ae6327e60ec0e0" candidate_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca3300cf09c04d6cac79551a5ab409ea"
I20240502 08:04:53.433259 12683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1abb47c8c0a642b097ae6327e60ec0e0" candidate_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b79fbd9f226f4270a50e10dce97bafad"
I20240502 08:04:53.433565 12609 raft_consensus.cc:3037] T 1abb47c8c0a642b097ae6327e60ec0e0 P ca3300cf09c04d6cac79551a5ab409ea [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:53.433820 12683 raft_consensus.cc:3037] T 1abb47c8c0a642b097ae6327e60ec0e0 P b79fbd9f226f4270a50e10dce97bafad [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:53.440376 12683 raft_consensus.cc:2454] T 1abb47c8c0a642b097ae6327e60ec0e0 P b79fbd9f226f4270a50e10dce97bafad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e739706a3cb4269b73b3dc63e28ccfe in term 1.
I20240502 08:04:53.440376 12609 raft_consensus.cc:2454] T 1abb47c8c0a642b097ae6327e60ec0e0 P ca3300cf09c04d6cac79551a5ab409ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e739706a3cb4269b73b3dc63e28ccfe in term 1.
I20240502 08:04:53.441730 12499 leader_election.cc:302] T 1abb47c8c0a642b097ae6327e60ec0e0 P 7e739706a3cb4269b73b3dc63e28ccfe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e739706a3cb4269b73b3dc63e28ccfe, b79fbd9f226f4270a50e10dce97bafad; no voters:
I20240502 08:04:53.442435 12731 raft_consensus.cc:2781] T 1abb47c8c0a642b097ae6327e60ec0e0 P 7e739706a3cb4269b73b3dc63e28ccfe [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:04:53.443490 12731 raft_consensus.cc:686] T 1abb47c8c0a642b097ae6327e60ec0e0 P 7e739706a3cb4269b73b3dc63e28ccfe [term 1 LEADER]: Becoming Leader. State: Replica: 7e739706a3cb4269b73b3dc63e28ccfe, State: Running, Role: LEADER
I20240502 08:04:53.444566 12731 consensus_queue.cc:237] T 1abb47c8c0a642b097ae6327e60ec0e0 P 7e739706a3cb4269b73b3dc63e28ccfe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:53.451488 12430 catalog_manager.cc:5484] T 1abb47c8c0a642b097ae6327e60ec0e0 P 7e739706a3cb4269b73b3dc63e28ccfe reported cstate change: term changed from 0 to 1, leader changed from <none> to 7e739706a3cb4269b73b3dc63e28ccfe (127.6.204.193). New cstate: current_term: 1 leader_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } health_report { overall_health: UNKNOWN } } }
I20240502 08:04:53.538944 12735 raft_consensus.cc:482] T 90b6d92c72f44d89a54c7b1b55470f4b P ca3300cf09c04d6cac79551a5ab409ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:04:53.539440 12735 raft_consensus.cc:504] T 90b6d92c72f44d89a54c7b1b55470f4b P ca3300cf09c04d6cac79551a5ab409ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:53.541000 12735 leader_election.cc:288] T 90b6d92c72f44d89a54c7b1b55470f4b P ca3300cf09c04d6cac79551a5ab409ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e739706a3cb4269b73b3dc63e28ccfe (127.6.204.193:32777), b79fbd9f226f4270a50e10dce97bafad (127.6.204.195:46407)
I20240502 08:04:53.541891 12534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b6d92c72f44d89a54c7b1b55470f4b" candidate_uuid: "ca3300cf09c04d6cac79551a5ab409ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" is_pre_election: true
I20240502 08:04:53.541945 12683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b6d92c72f44d89a54c7b1b55470f4b" candidate_uuid: "ca3300cf09c04d6cac79551a5ab409ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b79fbd9f226f4270a50e10dce97bafad" is_pre_election: true
I20240502 08:04:53.542587 12534 raft_consensus.cc:2454] T 90b6d92c72f44d89a54c7b1b55470f4b P 7e739706a3cb4269b73b3dc63e28ccfe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca3300cf09c04d6cac79551a5ab409ea in term 0.
I20240502 08:04:53.542688 12683 raft_consensus.cc:2454] T 90b6d92c72f44d89a54c7b1b55470f4b P b79fbd9f226f4270a50e10dce97bafad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca3300cf09c04d6cac79551a5ab409ea in term 0.
I20240502 08:04:53.543772 12573 leader_election.cc:302] T 90b6d92c72f44d89a54c7b1b55470f4b P ca3300cf09c04d6cac79551a5ab409ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e739706a3cb4269b73b3dc63e28ccfe, ca3300cf09c04d6cac79551a5ab409ea; no voters:
I20240502 08:04:53.544549 12735 raft_consensus.cc:2781] T 90b6d92c72f44d89a54c7b1b55470f4b P ca3300cf09c04d6cac79551a5ab409ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:04:53.544910 12735 raft_consensus.cc:482] T 90b6d92c72f44d89a54c7b1b55470f4b P ca3300cf09c04d6cac79551a5ab409ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:04:53.545193 12735 raft_consensus.cc:3037] T 90b6d92c72f44d89a54c7b1b55470f4b P ca3300cf09c04d6cac79551a5ab409ea [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:53.551349 12735 raft_consensus.cc:504] T 90b6d92c72f44d89a54c7b1b55470f4b P ca3300cf09c04d6cac79551a5ab409ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:53.552946 12735 leader_election.cc:288] T 90b6d92c72f44d89a54c7b1b55470f4b P ca3300cf09c04d6cac79551a5ab409ea [CANDIDATE]: Term 1 election: Requested vote from peers 7e739706a3cb4269b73b3dc63e28ccfe (127.6.204.193:32777), b79fbd9f226f4270a50e10dce97bafad (127.6.204.195:46407)
I20240502 08:04:53.553839 12534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b6d92c72f44d89a54c7b1b55470f4b" candidate_uuid: "ca3300cf09c04d6cac79551a5ab409ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e739706a3cb4269b73b3dc63e28ccfe"
I20240502 08:04:53.554024 12683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b6d92c72f44d89a54c7b1b55470f4b" candidate_uuid: "ca3300cf09c04d6cac79551a5ab409ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b79fbd9f226f4270a50e10dce97bafad"
I20240502 08:04:53.554535 12534 raft_consensus.cc:3037] T 90b6d92c72f44d89a54c7b1b55470f4b P 7e739706a3cb4269b73b3dc63e28ccfe [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:53.554626 12683 raft_consensus.cc:3037] T 90b6d92c72f44d89a54c7b1b55470f4b P b79fbd9f226f4270a50e10dce97bafad [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:53.561589 12534 raft_consensus.cc:2454] T 90b6d92c72f44d89a54c7b1b55470f4b P 7e739706a3cb4269b73b3dc63e28ccfe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca3300cf09c04d6cac79551a5ab409ea in term 1.
I20240502 08:04:53.561589 12683 raft_consensus.cc:2454] T 90b6d92c72f44d89a54c7b1b55470f4b P b79fbd9f226f4270a50e10dce97bafad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca3300cf09c04d6cac79551a5ab409ea in term 1.
I20240502 08:04:53.562953 12573 leader_election.cc:302] T 90b6d92c72f44d89a54c7b1b55470f4b P ca3300cf09c04d6cac79551a5ab409ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e739706a3cb4269b73b3dc63e28ccfe, ca3300cf09c04d6cac79551a5ab409ea; no voters:
I20240502 08:04:53.563647 12735 raft_consensus.cc:2781] T 90b6d92c72f44d89a54c7b1b55470f4b P ca3300cf09c04d6cac79551a5ab409ea [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:04:53.564577 12735 raft_consensus.cc:686] T 90b6d92c72f44d89a54c7b1b55470f4b P ca3300cf09c04d6cac79551a5ab409ea [term 1 LEADER]: Becoming Leader. State: Replica: ca3300cf09c04d6cac79551a5ab409ea, State: Running, Role: LEADER
I20240502 08:04:53.565277 12735 consensus_queue.cc:237] T 90b6d92c72f44d89a54c7b1b55470f4b P ca3300cf09c04d6cac79551a5ab409ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:53.572723 12430 catalog_manager.cc:5484] T 90b6d92c72f44d89a54c7b1b55470f4b P ca3300cf09c04d6cac79551a5ab409ea reported cstate change: term changed from 0 to 1, leader changed from <none> to ca3300cf09c04d6cac79551a5ab409ea (127.6.204.194). New cstate: current_term: 1 leader_uuid: "ca3300cf09c04d6cac79551a5ab409ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } health_report { overall_health: UNKNOWN } } }
I20240502 08:04:53.588272 6963 test_util.cc:268] Using random seed: 249679111
I20240502 08:04:53.610579 12430 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47766:
name: "dugtrio"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:04:53.612658 12430 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table dugtrio in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:04:53.631332 12480 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240502 08:04:53.641336 12521 tablet_service.cc:1467] Processing CreateTablet for tablet 0f00e1259b934b7a904dbc7dc4a434ae (DEFAULT_TABLE table=dugtrio [id=d58d27d9f8724a5ca12188de828572b7]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240502 08:04:53.642601 12521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f00e1259b934b7a904dbc7dc4a434ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:53.643203 12597 tablet_service.cc:1467] Processing CreateTablet for tablet 0f00e1259b934b7a904dbc7dc4a434ae (DEFAULT_TABLE table=dugtrio [id=d58d27d9f8724a5ca12188de828572b7]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240502 08:04:53.643926 12670 tablet_service.cc:1467] Processing CreateTablet for tablet 0f00e1259b934b7a904dbc7dc4a434ae (DEFAULT_TABLE table=dugtrio [id=d58d27d9f8724a5ca12188de828572b7]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240502 08:04:53.644425 12597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f00e1259b934b7a904dbc7dc4a434ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:53.645083 12670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f00e1259b934b7a904dbc7dc4a434ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:53.647199 12671 tablet_service.cc:1467] Processing CreateTablet for tablet bb44f0c8f9164047ae9d2ce1b67ab9da (DEFAULT_TABLE table=dugtrio [id=d58d27d9f8724a5ca12188de828572b7]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240502 08:04:53.648412 12671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb44f0c8f9164047ae9d2ce1b67ab9da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:53.649693 12673 tablet_service.cc:1467] Processing CreateTablet for tablet 9badc171df9a47f9bb33b775dea31604 (DEFAULT_TABLE table=dugtrio [id=d58d27d9f8724a5ca12188de828572b7]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240502 08:04:53.650949 12673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9badc171df9a47f9bb33b775dea31604. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:53.651623 12596 tablet_service.cc:1467] Processing CreateTablet for tablet bb44f0c8f9164047ae9d2ce1b67ab9da (DEFAULT_TABLE table=dugtrio [id=d58d27d9f8724a5ca12188de828572b7]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240502 08:04:53.652827 12596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb44f0c8f9164047ae9d2ce1b67ab9da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:53.652854 12672 tablet_service.cc:1467] Processing CreateTablet for tablet 212a1cd8d6244f73b013888b748c3ee8 (DEFAULT_TABLE table=dugtrio [id=d58d27d9f8724a5ca12188de828572b7]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240502 08:04:53.654318 12672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 212a1cd8d6244f73b013888b748c3ee8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:53.659576 12524 tablet_service.cc:1467] Processing CreateTablet for tablet bb44f0c8f9164047ae9d2ce1b67ab9da (DEFAULT_TABLE table=dugtrio [id=d58d27d9f8724a5ca12188de828572b7]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240502 08:04:53.663188 12599 tablet_service.cc:1467] Processing CreateTablet for tablet 9badc171df9a47f9bb33b775dea31604 (DEFAULT_TABLE table=dugtrio [id=d58d27d9f8724a5ca12188de828572b7]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240502 08:04:53.663460 12522 tablet_service.cc:1467] Processing CreateTablet for tablet 212a1cd8d6244f73b013888b748c3ee8 (DEFAULT_TABLE table=dugtrio [id=d58d27d9f8724a5ca12188de828572b7]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240502 08:04:53.664497 12524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb44f0c8f9164047ae9d2ce1b67ab9da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:53.665604 12598 tablet_service.cc:1467] Processing CreateTablet for tablet 212a1cd8d6244f73b013888b748c3ee8 (DEFAULT_TABLE table=dugtrio [id=d58d27d9f8724a5ca12188de828572b7]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240502 08:04:53.663180 12523 tablet_service.cc:1467] Processing CreateTablet for tablet 9badc171df9a47f9bb33b775dea31604 (DEFAULT_TABLE table=dugtrio [id=d58d27d9f8724a5ca12188de828572b7]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240502 08:04:53.667917 12523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9badc171df9a47f9bb33b775dea31604. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:53.670958 12522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 212a1cd8d6244f73b013888b748c3ee8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:53.664498 12599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9badc171df9a47f9bb33b775dea31604. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:53.672849 12598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 212a1cd8d6244f73b013888b748c3ee8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:04:53.678213 12734 tablet_bootstrap.cc:492] T 9badc171df9a47f9bb33b775dea31604 P b79fbd9f226f4270a50e10dce97bafad: Bootstrap starting.
I20240502 08:04:53.680589 12728 tablet_bootstrap.cc:492] T 0f00e1259b934b7a904dbc7dc4a434ae P 7e739706a3cb4269b73b3dc63e28ccfe: Bootstrap starting.
I20240502 08:04:53.683565 12734 tablet_bootstrap.cc:654] T 9badc171df9a47f9bb33b775dea31604 P b79fbd9f226f4270a50e10dce97bafad: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:53.688004 12728 tablet_bootstrap.cc:654] T 0f00e1259b934b7a904dbc7dc4a434ae P 7e739706a3cb4269b73b3dc63e28ccfe: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:53.693389 12730 tablet_bootstrap.cc:492] T 212a1cd8d6244f73b013888b748c3ee8 P ca3300cf09c04d6cac79551a5ab409ea: Bootstrap starting.
I20240502 08:04:53.701956 12730 tablet_bootstrap.cc:654] T 212a1cd8d6244f73b013888b748c3ee8 P ca3300cf09c04d6cac79551a5ab409ea: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:53.714690 12734 tablet_bootstrap.cc:492] T 9badc171df9a47f9bb33b775dea31604 P b79fbd9f226f4270a50e10dce97bafad: No bootstrap required, opened a new log
I20240502 08:04:53.715292 12734 ts_tablet_manager.cc:1367] T 9badc171df9a47f9bb33b775dea31604 P b79fbd9f226f4270a50e10dce97bafad: Time spent bootstrapping tablet: real 0.037s user 0.013s sys 0.003s
I20240502 08:04:53.718319 12728 tablet_bootstrap.cc:492] T 0f00e1259b934b7a904dbc7dc4a434ae P 7e739706a3cb4269b73b3dc63e28ccfe: No bootstrap required, opened a new log
I20240502 08:04:53.718371 12730 tablet_bootstrap.cc:492] T 212a1cd8d6244f73b013888b748c3ee8 P ca3300cf09c04d6cac79551a5ab409ea: No bootstrap required, opened a new log
I20240502 08:04:53.717988 12734 raft_consensus.cc:348] T 9badc171df9a47f9bb33b775dea31604 P b79fbd9f226f4270a50e10dce97bafad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:53.718919 12728 ts_tablet_manager.cc:1367] T 0f00e1259b934b7a904dbc7dc4a434ae P 7e739706a3cb4269b73b3dc63e28ccfe: Time spent bootstrapping tablet: real 0.039s user 0.007s sys 0.003s
I20240502 08:04:53.719177 12734 raft_consensus.cc:374] T 9badc171df9a47f9bb33b775dea31604 P b79fbd9f226f4270a50e10dce97bafad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:53.719182 12730 ts_tablet_manager.cc:1367] T 212a1cd8d6244f73b013888b748c3ee8 P ca3300cf09c04d6cac79551a5ab409ea: Time spent bootstrapping tablet: real 0.026s user 0.007s sys 0.008s
I20240502 08:04:53.721774 12734 raft_consensus.cc:729] T 9badc171df9a47f9bb33b775dea31604 P b79fbd9f226f4270a50e10dce97bafad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b79fbd9f226f4270a50e10dce97bafad, State: Initialized, Role: FOLLOWER
I20240502 08:04:53.722471 12728 raft_consensus.cc:348] T 0f00e1259b934b7a904dbc7dc4a434ae P 7e739706a3cb4269b73b3dc63e28ccfe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:53.724359 12728 raft_consensus.cc:374] T 0f00e1259b934b7a904dbc7dc4a434ae P 7e739706a3cb4269b73b3dc63e28ccfe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:53.724668 12728 raft_consensus.cc:729] T 0f00e1259b934b7a904dbc7dc4a434ae P 7e739706a3cb4269b73b3dc63e28ccfe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e739706a3cb4269b73b3dc63e28ccfe, State: Initialized, Role: FOLLOWER
I20240502 08:04:53.725899 12734 consensus_queue.cc:260] T 9badc171df9a47f9bb33b775dea31604 P b79fbd9f226f4270a50e10dce97bafad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:53.728317 12734 ts_tablet_manager.cc:1398] T 9badc171df9a47f9bb33b775dea31604 P b79fbd9f226f4270a50e10dce97bafad: Time spent starting tablet: real 0.013s user 0.005s sys 0.000s
I20240502 08:04:53.729296 12734 tablet_bootstrap.cc:492] T 0f00e1259b934b7a904dbc7dc4a434ae P b79fbd9f226f4270a50e10dce97bafad: Bootstrap starting.
I20240502 08:04:53.725363 12728 consensus_queue.cc:260] T 0f00e1259b934b7a904dbc7dc4a434ae P 7e739706a3cb4269b73b3dc63e28ccfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:53.738828 12728 ts_tablet_manager.cc:1398] T 0f00e1259b934b7a904dbc7dc4a434ae P 7e739706a3cb4269b73b3dc63e28ccfe: Time spent starting tablet: real 0.020s user 0.006s sys 0.000s
I20240502 08:04:53.739902 12728 tablet_bootstrap.cc:492] T 9badc171df9a47f9bb33b775dea31604 P 7e739706a3cb4269b73b3dc63e28ccfe: Bootstrap starting.
I20240502 08:04:53.745563 12728 tablet_bootstrap.cc:654] T 9badc171df9a47f9bb33b775dea31604 P 7e739706a3cb4269b73b3dc63e28ccfe: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:53.746699 12734 tablet_bootstrap.cc:654] T 0f00e1259b934b7a904dbc7dc4a434ae P b79fbd9f226f4270a50e10dce97bafad: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:53.723202 12730 raft_consensus.cc:348] T 212a1cd8d6244f73b013888b748c3ee8 P ca3300cf09c04d6cac79551a5ab409ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:53.749969 12730 raft_consensus.cc:374] T 212a1cd8d6244f73b013888b748c3ee8 P ca3300cf09c04d6cac79551a5ab409ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:53.750320 12730 raft_consensus.cc:729] T 212a1cd8d6244f73b013888b748c3ee8 P ca3300cf09c04d6cac79551a5ab409ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca3300cf09c04d6cac79551a5ab409ea, State: Initialized, Role: FOLLOWER
I20240502 08:04:53.751075 12730 consensus_queue.cc:260] T 212a1cd8d6244f73b013888b748c3ee8 P ca3300cf09c04d6cac79551a5ab409ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:53.758795 12730 ts_tablet_manager.cc:1398] T 212a1cd8d6244f73b013888b748c3ee8 P ca3300cf09c04d6cac79551a5ab409ea: Time spent starting tablet: real 0.039s user 0.006s sys 0.000s
I20240502 08:04:53.760035 12730 tablet_bootstrap.cc:492] T bb44f0c8f9164047ae9d2ce1b67ab9da P ca3300cf09c04d6cac79551a5ab409ea: Bootstrap starting.
I20240502 08:04:53.769465 12734 tablet_bootstrap.cc:492] T 0f00e1259b934b7a904dbc7dc4a434ae P b79fbd9f226f4270a50e10dce97bafad: No bootstrap required, opened a new log
I20240502 08:04:53.770174 12734 ts_tablet_manager.cc:1367] T 0f00e1259b934b7a904dbc7dc4a434ae P b79fbd9f226f4270a50e10dce97bafad: Time spent bootstrapping tablet: real 0.041s user 0.011s sys 0.005s
I20240502 08:04:53.775435 12730 tablet_bootstrap.cc:654] T bb44f0c8f9164047ae9d2ce1b67ab9da P ca3300cf09c04d6cac79551a5ab409ea: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:53.776830 12728 tablet_bootstrap.cc:492] T 9badc171df9a47f9bb33b775dea31604 P 7e739706a3cb4269b73b3dc63e28ccfe: No bootstrap required, opened a new log
I20240502 08:04:53.777345 12728 ts_tablet_manager.cc:1367] T 9badc171df9a47f9bb33b775dea31604 P 7e739706a3cb4269b73b3dc63e28ccfe: Time spent bootstrapping tablet: real 0.038s user 0.017s sys 0.007s
I20240502 08:04:53.773589 12734 raft_consensus.cc:348] T 0f00e1259b934b7a904dbc7dc4a434ae P b79fbd9f226f4270a50e10dce97bafad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:53.778506 12734 raft_consensus.cc:374] T 0f00e1259b934b7a904dbc7dc4a434ae P b79fbd9f226f4270a50e10dce97bafad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:53.778832 12734 raft_consensus.cc:729] T 0f00e1259b934b7a904dbc7dc4a434ae P b79fbd9f226f4270a50e10dce97bafad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b79fbd9f226f4270a50e10dce97bafad, State: Initialized, Role: FOLLOWER
I20240502 08:04:53.779582 12734 consensus_queue.cc:260] T 0f00e1259b934b7a904dbc7dc4a434ae P b79fbd9f226f4270a50e10dce97bafad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:53.780117 12728 raft_consensus.cc:348] T 9badc171df9a47f9bb33b775dea31604 P 7e739706a3cb4269b73b3dc63e28ccfe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:53.780838 12728 raft_consensus.cc:374] T 9badc171df9a47f9bb33b775dea31604 P 7e739706a3cb4269b73b3dc63e28ccfe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:53.781121 12728 raft_consensus.cc:729] T 9badc171df9a47f9bb33b775dea31604 P 7e739706a3cb4269b73b3dc63e28ccfe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e739706a3cb4269b73b3dc63e28ccfe, State: Initialized, Role: FOLLOWER
I20240502 08:04:53.782228 12734 ts_tablet_manager.cc:1398] T 0f00e1259b934b7a904dbc7dc4a434ae P b79fbd9f226f4270a50e10dce97bafad: Time spent starting tablet: real 0.012s user 0.005s sys 0.003s
I20240502 08:04:53.781883 12728 consensus_queue.cc:260] T 9badc171df9a47f9bb33b775dea31604 P 7e739706a3cb4269b73b3dc63e28ccfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:53.784657 12728 ts_tablet_manager.cc:1398] T 9badc171df9a47f9bb33b775dea31604 P 7e739706a3cb4269b73b3dc63e28ccfe: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20240502 08:04:53.785387 12734 tablet_bootstrap.cc:492] T bb44f0c8f9164047ae9d2ce1b67ab9da P b79fbd9f226f4270a50e10dce97bafad: Bootstrap starting.
I20240502 08:04:53.785792 12728 tablet_bootstrap.cc:492] T bb44f0c8f9164047ae9d2ce1b67ab9da P 7e739706a3cb4269b73b3dc63e28ccfe: Bootstrap starting.
I20240502 08:04:53.791738 12728 tablet_bootstrap.cc:654] T bb44f0c8f9164047ae9d2ce1b67ab9da P 7e739706a3cb4269b73b3dc63e28ccfe: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:53.791970 12730 tablet_bootstrap.cc:492] T bb44f0c8f9164047ae9d2ce1b67ab9da P ca3300cf09c04d6cac79551a5ab409ea: No bootstrap required, opened a new log
I20240502 08:04:53.792439 12730 ts_tablet_manager.cc:1367] T bb44f0c8f9164047ae9d2ce1b67ab9da P ca3300cf09c04d6cac79551a5ab409ea: Time spent bootstrapping tablet: real 0.033s user 0.010s sys 0.000s
I20240502 08:04:53.795390 12730 raft_consensus.cc:348] T bb44f0c8f9164047ae9d2ce1b67ab9da P ca3300cf09c04d6cac79551a5ab409ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:53.796296 12730 raft_consensus.cc:374] T bb44f0c8f9164047ae9d2ce1b67ab9da P ca3300cf09c04d6cac79551a5ab409ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:53.796584 12730 raft_consensus.cc:729] T bb44f0c8f9164047ae9d2ce1b67ab9da P ca3300cf09c04d6cac79551a5ab409ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca3300cf09c04d6cac79551a5ab409ea, State: Initialized, Role: FOLLOWER
I20240502 08:04:53.797289 12730 consensus_queue.cc:260] T bb44f0c8f9164047ae9d2ce1b67ab9da P ca3300cf09c04d6cac79551a5ab409ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:53.798383 12728 tablet_bootstrap.cc:492] T bb44f0c8f9164047ae9d2ce1b67ab9da P 7e739706a3cb4269b73b3dc63e28ccfe: No bootstrap required, opened a new log
I20240502 08:04:53.798871 12728 ts_tablet_manager.cc:1367] T bb44f0c8f9164047ae9d2ce1b67ab9da P 7e739706a3cb4269b73b3dc63e28ccfe: Time spent bootstrapping tablet: real 0.013s user 0.012s sys 0.000s
I20240502 08:04:53.799907 12734 tablet_bootstrap.cc:654] T bb44f0c8f9164047ae9d2ce1b67ab9da P b79fbd9f226f4270a50e10dce97bafad: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:53.800655 12730 ts_tablet_manager.cc:1398] T bb44f0c8f9164047ae9d2ce1b67ab9da P ca3300cf09c04d6cac79551a5ab409ea: Time spent starting tablet: real 0.008s user 0.009s sys 0.000s
I20240502 08:04:53.801731 12730 tablet_bootstrap.cc:492] T 9badc171df9a47f9bb33b775dea31604 P ca3300cf09c04d6cac79551a5ab409ea: Bootstrap starting.
I20240502 08:04:53.801537 12728 raft_consensus.cc:348] T bb44f0c8f9164047ae9d2ce1b67ab9da P 7e739706a3cb4269b73b3dc63e28ccfe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:53.802373 12728 raft_consensus.cc:374] T bb44f0c8f9164047ae9d2ce1b67ab9da P 7e739706a3cb4269b73b3dc63e28ccfe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:53.802757 12728 raft_consensus.cc:729] T bb44f0c8f9164047ae9d2ce1b67ab9da P 7e739706a3cb4269b73b3dc63e28ccfe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e739706a3cb4269b73b3dc63e28ccfe, State: Initialized, Role: FOLLOWER
I20240502 08:04:53.803740 12728 consensus_queue.cc:260] T bb44f0c8f9164047ae9d2ce1b67ab9da P 7e739706a3cb4269b73b3dc63e28ccfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:53.807246 12728 ts_tablet_manager.cc:1398] T bb44f0c8f9164047ae9d2ce1b67ab9da P 7e739706a3cb4269b73b3dc63e28ccfe: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20240502 08:04:53.808467 12728 tablet_bootstrap.cc:492] T 212a1cd8d6244f73b013888b748c3ee8 P 7e739706a3cb4269b73b3dc63e28ccfe: Bootstrap starting.
I20240502 08:04:53.811475 12734 tablet_bootstrap.cc:492] T bb44f0c8f9164047ae9d2ce1b67ab9da P b79fbd9f226f4270a50e10dce97bafad: No bootstrap required, opened a new log
I20240502 08:04:53.812014 12734 ts_tablet_manager.cc:1367] T bb44f0c8f9164047ae9d2ce1b67ab9da P b79fbd9f226f4270a50e10dce97bafad: Time spent bootstrapping tablet: real 0.027s user 0.012s sys 0.003s
I20240502 08:04:53.812180 12730 tablet_bootstrap.cc:654] T 9badc171df9a47f9bb33b775dea31604 P ca3300cf09c04d6cac79551a5ab409ea: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:53.814066 12728 tablet_bootstrap.cc:654] T 212a1cd8d6244f73b013888b748c3ee8 P 7e739706a3cb4269b73b3dc63e28ccfe: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:53.814919 12734 raft_consensus.cc:348] T bb44f0c8f9164047ae9d2ce1b67ab9da P b79fbd9f226f4270a50e10dce97bafad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:53.815721 12734 raft_consensus.cc:374] T bb44f0c8f9164047ae9d2ce1b67ab9da P b79fbd9f226f4270a50e10dce97bafad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:53.816071 12734 raft_consensus.cc:729] T bb44f0c8f9164047ae9d2ce1b67ab9da P b79fbd9f226f4270a50e10dce97bafad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b79fbd9f226f4270a50e10dce97bafad, State: Initialized, Role: FOLLOWER
I20240502 08:04:53.816948 12734 consensus_queue.cc:260] T bb44f0c8f9164047ae9d2ce1b67ab9da P b79fbd9f226f4270a50e10dce97bafad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:53.822399 12728 tablet_bootstrap.cc:492] T 212a1cd8d6244f73b013888b748c3ee8 P 7e739706a3cb4269b73b3dc63e28ccfe: No bootstrap required, opened a new log
I20240502 08:04:53.822927 12728 ts_tablet_manager.cc:1367] T 212a1cd8d6244f73b013888b748c3ee8 P 7e739706a3cb4269b73b3dc63e28ccfe: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.004s
I20240502 08:04:53.823014 12734 ts_tablet_manager.cc:1398] T bb44f0c8f9164047ae9d2ce1b67ab9da P b79fbd9f226f4270a50e10dce97bafad: Time spent starting tablet: real 0.011s user 0.006s sys 0.003s
I20240502 08:04:53.824067 12734 tablet_bootstrap.cc:492] T 212a1cd8d6244f73b013888b748c3ee8 P b79fbd9f226f4270a50e10dce97bafad: Bootstrap starting.
I20240502 08:04:53.825668 12728 raft_consensus.cc:348] T 212a1cd8d6244f73b013888b748c3ee8 P 7e739706a3cb4269b73b3dc63e28ccfe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:53.829921 12730 tablet_bootstrap.cc:492] T 9badc171df9a47f9bb33b775dea31604 P ca3300cf09c04d6cac79551a5ab409ea: No bootstrap required, opened a new log
I20240502 08:04:53.830083 12734 tablet_bootstrap.cc:654] T 212a1cd8d6244f73b013888b748c3ee8 P b79fbd9f226f4270a50e10dce97bafad: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:53.830488 12730 ts_tablet_manager.cc:1367] T 9badc171df9a47f9bb33b775dea31604 P ca3300cf09c04d6cac79551a5ab409ea: Time spent bootstrapping tablet: real 0.029s user 0.012s sys 0.007s
I20240502 08:04:53.828020 12728 raft_consensus.cc:374] T 212a1cd8d6244f73b013888b748c3ee8 P 7e739706a3cb4269b73b3dc63e28ccfe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:53.832209 12728 raft_consensus.cc:729] T 212a1cd8d6244f73b013888b748c3ee8 P 7e739706a3cb4269b73b3dc63e28ccfe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e739706a3cb4269b73b3dc63e28ccfe, State: Initialized, Role: FOLLOWER
I20240502 08:04:53.833020 12728 consensus_queue.cc:260] T 212a1cd8d6244f73b013888b748c3ee8 P 7e739706a3cb4269b73b3dc63e28ccfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:53.833333 12730 raft_consensus.cc:348] T 9badc171df9a47f9bb33b775dea31604 P ca3300cf09c04d6cac79551a5ab409ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:53.834201 12730 raft_consensus.cc:374] T 9badc171df9a47f9bb33b775dea31604 P ca3300cf09c04d6cac79551a5ab409ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:53.834530 12730 raft_consensus.cc:729] T 9badc171df9a47f9bb33b775dea31604 P ca3300cf09c04d6cac79551a5ab409ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca3300cf09c04d6cac79551a5ab409ea, State: Initialized, Role: FOLLOWER
I20240502 08:04:53.835080 12731 raft_consensus.cc:482] T bb44f0c8f9164047ae9d2ce1b67ab9da P 7e739706a3cb4269b73b3dc63e28ccfe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:04:53.835513 12730 consensus_queue.cc:260] T 9badc171df9a47f9bb33b775dea31604 P ca3300cf09c04d6cac79551a5ab409ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:53.835602 12731 raft_consensus.cc:504] T bb44f0c8f9164047ae9d2ce1b67ab9da P 7e739706a3cb4269b73b3dc63e28ccfe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:53.838053 12730 ts_tablet_manager.cc:1398] T 9badc171df9a47f9bb33b775dea31604 P ca3300cf09c04d6cac79551a5ab409ea: Time spent starting tablet: real 0.007s user 0.002s sys 0.004s
I20240502 08:04:53.839143 12730 tablet_bootstrap.cc:492] T 0f00e1259b934b7a904dbc7dc4a434ae P ca3300cf09c04d6cac79551a5ab409ea: Bootstrap starting.
I20240502 08:04:53.839780 12609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb44f0c8f9164047ae9d2ce1b67ab9da" candidate_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca3300cf09c04d6cac79551a5ab409ea" is_pre_election: true
I20240502 08:04:53.840065 12683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb44f0c8f9164047ae9d2ce1b67ab9da" candidate_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b79fbd9f226f4270a50e10dce97bafad" is_pre_election: true
I20240502 08:04:53.840476 12609 raft_consensus.cc:2454] T bb44f0c8f9164047ae9d2ce1b67ab9da P ca3300cf09c04d6cac79551a5ab409ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e739706a3cb4269b73b3dc63e28ccfe in term 0.
I20240502 08:04:53.840656 12683 raft_consensus.cc:2454] T bb44f0c8f9164047ae9d2ce1b67ab9da P b79fbd9f226f4270a50e10dce97bafad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e739706a3cb4269b73b3dc63e28ccfe in term 0.
I20240502 08:04:53.841603 12731 leader_election.cc:288] T bb44f0c8f9164047ae9d2ce1b67ab9da P 7e739706a3cb4269b73b3dc63e28ccfe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ca3300cf09c04d6cac79551a5ab409ea (127.6.204.194:39897), b79fbd9f226f4270a50e10dce97bafad (127.6.204.195:46407)
I20240502 08:04:53.841871 12498 leader_election.cc:302] T bb44f0c8f9164047ae9d2ce1b67ab9da P 7e739706a3cb4269b73b3dc63e28ccfe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e739706a3cb4269b73b3dc63e28ccfe, ca3300cf09c04d6cac79551a5ab409ea; no voters:
I20240502 08:04:53.844734 12731 raft_consensus.cc:2781] T bb44f0c8f9164047ae9d2ce1b67ab9da P 7e739706a3cb4269b73b3dc63e28ccfe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:04:53.835233 12728 ts_tablet_manager.cc:1398] T 212a1cd8d6244f73b013888b748c3ee8 P 7e739706a3cb4269b73b3dc63e28ccfe: Time spent starting tablet: real 0.012s user 0.006s sys 0.001s
I20240502 08:04:53.845114 12731 raft_consensus.cc:482] T bb44f0c8f9164047ae9d2ce1b67ab9da P 7e739706a3cb4269b73b3dc63e28ccfe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:04:53.845391 12730 tablet_bootstrap.cc:654] T 0f00e1259b934b7a904dbc7dc4a434ae P ca3300cf09c04d6cac79551a5ab409ea: Neither blocks nor log segments found. Creating new log.
I20240502 08:04:53.845489 12731 raft_consensus.cc:3037] T bb44f0c8f9164047ae9d2ce1b67ab9da P 7e739706a3cb4269b73b3dc63e28ccfe [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:53.851950 12734 tablet_bootstrap.cc:492] T 212a1cd8d6244f73b013888b748c3ee8 P b79fbd9f226f4270a50e10dce97bafad: No bootstrap required, opened a new log
I20240502 08:04:53.852519 12734 ts_tablet_manager.cc:1367] T 212a1cd8d6244f73b013888b748c3ee8 P b79fbd9f226f4270a50e10dce97bafad: Time spent bootstrapping tablet: real 0.029s user 0.015s sys 0.002s
I20240502 08:04:53.853370 12731 raft_consensus.cc:504] T bb44f0c8f9164047ae9d2ce1b67ab9da P 7e739706a3cb4269b73b3dc63e28ccfe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:53.855211 12734 raft_consensus.cc:348] T 212a1cd8d6244f73b013888b748c3ee8 P b79fbd9f226f4270a50e10dce97bafad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:53.856117 12734 raft_consensus.cc:374] T 212a1cd8d6244f73b013888b748c3ee8 P b79fbd9f226f4270a50e10dce97bafad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:53.856134 12609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb44f0c8f9164047ae9d2ce1b67ab9da" candidate_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca3300cf09c04d6cac79551a5ab409ea"
I20240502 08:04:53.856564 12734 raft_consensus.cc:729] T 212a1cd8d6244f73b013888b748c3ee8 P b79fbd9f226f4270a50e10dce97bafad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b79fbd9f226f4270a50e10dce97bafad, State: Initialized, Role: FOLLOWER
I20240502 08:04:53.856813 12609 raft_consensus.cc:3037] T bb44f0c8f9164047ae9d2ce1b67ab9da P ca3300cf09c04d6cac79551a5ab409ea [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:53.857785 12734 consensus_queue.cc:260] T 212a1cd8d6244f73b013888b748c3ee8 P b79fbd9f226f4270a50e10dce97bafad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:53.861145 12731 leader_election.cc:288] T bb44f0c8f9164047ae9d2ce1b67ab9da P 7e739706a3cb4269b73b3dc63e28ccfe [CANDIDATE]: Term 1 election: Requested vote from peers ca3300cf09c04d6cac79551a5ab409ea (127.6.204.194:39897), b79fbd9f226f4270a50e10dce97bafad (127.6.204.195:46407)
I20240502 08:04:53.862277 12683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb44f0c8f9164047ae9d2ce1b67ab9da" candidate_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b79fbd9f226f4270a50e10dce97bafad"
I20240502 08:04:53.862886 12683 raft_consensus.cc:3037] T bb44f0c8f9164047ae9d2ce1b67ab9da P b79fbd9f226f4270a50e10dce97bafad [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:53.863479 12750 consensus_queue.cc:1035] T bde9453a83b8464899af354b5e24806f P b79fbd9f226f4270a50e10dce97bafad [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 08:04:53.866171 12609 raft_consensus.cc:2454] T bb44f0c8f9164047ae9d2ce1b67ab9da P ca3300cf09c04d6cac79551a5ab409ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e739706a3cb4269b73b3dc63e28ccfe in term 1.
I20240502 08:04:53.861191 12755 raft_consensus.cc:482] T 0f00e1259b934b7a904dbc7dc4a434ae P 7e739706a3cb4269b73b3dc63e28ccfe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:04:53.867404 12498 leader_election.cc:302] T bb44f0c8f9164047ae9d2ce1b67ab9da P 7e739706a3cb4269b73b3dc63e28ccfe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e739706a3cb4269b73b3dc63e28ccfe, ca3300cf09c04d6cac79551a5ab409ea; no voters:
I20240502 08:04:53.867496 12755 raft_consensus.cc:504] T 0f00e1259b934b7a904dbc7dc4a434ae P 7e739706a3cb4269b73b3dc63e28ccfe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:53.869526 12755 leader_election.cc:288] T 0f00e1259b934b7a904dbc7dc4a434ae P 7e739706a3cb4269b73b3dc63e28ccfe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ca3300cf09c04d6cac79551a5ab409ea (127.6.204.194:39897), b79fbd9f226f4270a50e10dce97bafad (127.6.204.195:46407)
I20240502 08:04:53.869982 12755 raft_consensus.cc:2781] T bb44f0c8f9164047ae9d2ce1b67ab9da P 7e739706a3cb4269b73b3dc63e28ccfe [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:04:53.870633 12755 raft_consensus.cc:686] T bb44f0c8f9164047ae9d2ce1b67ab9da P 7e739706a3cb4269b73b3dc63e28ccfe [term 1 LEADER]: Becoming Leader. State: Replica: 7e739706a3cb4269b73b3dc63e28ccfe, State: Running, Role: LEADER
I20240502 08:04:53.871451 12755 consensus_queue.cc:237] T bb44f0c8f9164047ae9d2ce1b67ab9da P 7e739706a3cb4269b73b3dc63e28ccfe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:53.872841 12682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f00e1259b934b7a904dbc7dc4a434ae" candidate_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b79fbd9f226f4270a50e10dce97bafad" is_pre_election: true
I20240502 08:04:53.873507 12682 raft_consensus.cc:2454] T 0f00e1259b934b7a904dbc7dc4a434ae P b79fbd9f226f4270a50e10dce97bafad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e739706a3cb4269b73b3dc63e28ccfe in term 0.
I20240502 08:04:53.874679 12499 leader_election.cc:302] T 0f00e1259b934b7a904dbc7dc4a434ae P 7e739706a3cb4269b73b3dc63e28ccfe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e739706a3cb4269b73b3dc63e28ccfe, b79fbd9f226f4270a50e10dce97bafad; no voters:
I20240502 08:04:53.876776 12734 ts_tablet_manager.cc:1398] T 212a1cd8d6244f73b013888b748c3ee8 P b79fbd9f226f4270a50e10dce97bafad: Time spent starting tablet: real 0.024s user 0.005s sys 0.000s
I20240502 08:04:53.876677 12608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f00e1259b934b7a904dbc7dc4a434ae" candidate_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca3300cf09c04d6cac79551a5ab409ea" is_pre_election: true
W20240502 08:04:53.878387 12498 leader_election.cc:341] T 0f00e1259b934b7a904dbc7dc4a434ae P 7e739706a3cb4269b73b3dc63e28ccfe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ca3300cf09c04d6cac79551a5ab409ea (127.6.204.194:39897): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:04:53.883877 12739 consensus_queue.cc:1035] T 7fe7134eaf2e4e29ac762afe8855bdac P b79fbd9f226f4270a50e10dce97bafad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:04:53.886273 12683 raft_consensus.cc:2454] T bb44f0c8f9164047ae9d2ce1b67ab9da P b79fbd9f226f4270a50e10dce97bafad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e739706a3cb4269b73b3dc63e28ccfe in term 1.
I20240502 08:04:53.888031 12731 raft_consensus.cc:2781] T 0f00e1259b934b7a904dbc7dc4a434ae P 7e739706a3cb4269b73b3dc63e28ccfe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:04:53.890161 12731 raft_consensus.cc:482] T 0f00e1259b934b7a904dbc7dc4a434ae P 7e739706a3cb4269b73b3dc63e28ccfe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:04:53.890580 12731 raft_consensus.cc:3037] T 0f00e1259b934b7a904dbc7dc4a434ae P 7e739706a3cb4269b73b3dc63e28ccfe [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:53.893114 12730 tablet_bootstrap.cc:492] T 0f00e1259b934b7a904dbc7dc4a434ae P ca3300cf09c04d6cac79551a5ab409ea: No bootstrap required, opened a new log
I20240502 08:04:53.893707 12730 ts_tablet_manager.cc:1367] T 0f00e1259b934b7a904dbc7dc4a434ae P ca3300cf09c04d6cac79551a5ab409ea: Time spent bootstrapping tablet: real 0.055s user 0.024s sys 0.021s
I20240502 08:04:53.897147 12730 raft_consensus.cc:348] T 0f00e1259b934b7a904dbc7dc4a434ae P ca3300cf09c04d6cac79551a5ab409ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:53.898087 12730 raft_consensus.cc:374] T 0f00e1259b934b7a904dbc7dc4a434ae P ca3300cf09c04d6cac79551a5ab409ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:04:53.898455 12730 raft_consensus.cc:729] T 0f00e1259b934b7a904dbc7dc4a434ae P ca3300cf09c04d6cac79551a5ab409ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca3300cf09c04d6cac79551a5ab409ea, State: Initialized, Role: FOLLOWER
I20240502 08:04:53.899453 12730 consensus_queue.cc:260] T 0f00e1259b934b7a904dbc7dc4a434ae P ca3300cf09c04d6cac79551a5ab409ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:53.902449 12730 ts_tablet_manager.cc:1398] T 0f00e1259b934b7a904dbc7dc4a434ae P ca3300cf09c04d6cac79551a5ab409ea: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20240502 08:04:53.905230 12731 raft_consensus.cc:504] T 0f00e1259b934b7a904dbc7dc4a434ae P 7e739706a3cb4269b73b3dc63e28ccfe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:53.908061 12608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f00e1259b934b7a904dbc7dc4a434ae" candidate_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca3300cf09c04d6cac79551a5ab409ea"
I20240502 08:04:53.908684 12608 raft_consensus.cc:3037] T 0f00e1259b934b7a904dbc7dc4a434ae P ca3300cf09c04d6cac79551a5ab409ea [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:53.923007 12683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f00e1259b934b7a904dbc7dc4a434ae" candidate_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b79fbd9f226f4270a50e10dce97bafad"
I20240502 08:04:53.923722 12683 raft_consensus.cc:3037] T 0f00e1259b934b7a904dbc7dc4a434ae P b79fbd9f226f4270a50e10dce97bafad [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:53.924960 12608 raft_consensus.cc:2454] T 0f00e1259b934b7a904dbc7dc4a434ae P ca3300cf09c04d6cac79551a5ab409ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e739706a3cb4269b73b3dc63e28ccfe in term 1.
I20240502 08:04:53.926622 12498 leader_election.cc:302] T 0f00e1259b934b7a904dbc7dc4a434ae P 7e739706a3cb4269b73b3dc63e28ccfe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e739706a3cb4269b73b3dc63e28ccfe, ca3300cf09c04d6cac79551a5ab409ea; no voters:
I20240502 08:04:53.927474 12755 raft_consensus.cc:2781] T 0f00e1259b934b7a904dbc7dc4a434ae P 7e739706a3cb4269b73b3dc63e28ccfe [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:04:53.928138 12755 raft_consensus.cc:686] T 0f00e1259b934b7a904dbc7dc4a434ae P 7e739706a3cb4269b73b3dc63e28ccfe [term 1 LEADER]: Becoming Leader. State: Replica: 7e739706a3cb4269b73b3dc63e28ccfe, State: Running, Role: LEADER
I20240502 08:04:53.929500 12780 raft_consensus.cc:482] T 212a1cd8d6244f73b013888b748c3ee8 P 7e739706a3cb4269b73b3dc63e28ccfe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:04:53.929072 12755 consensus_queue.cc:237] T 0f00e1259b934b7a904dbc7dc4a434ae P 7e739706a3cb4269b73b3dc63e28ccfe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:53.929948 12780 raft_consensus.cc:504] T 212a1cd8d6244f73b013888b748c3ee8 P 7e739706a3cb4269b73b3dc63e28ccfe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:53.931236 12731 leader_election.cc:288] T 0f00e1259b934b7a904dbc7dc4a434ae P 7e739706a3cb4269b73b3dc63e28ccfe [CANDIDATE]: Term 1 election: Requested vote from peers ca3300cf09c04d6cac79551a5ab409ea (127.6.204.194:39897), b79fbd9f226f4270a50e10dce97bafad (127.6.204.195:46407)
I20240502 08:04:53.934789 12780 leader_election.cc:288] T 212a1cd8d6244f73b013888b748c3ee8 P 7e739706a3cb4269b73b3dc63e28ccfe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ca3300cf09c04d6cac79551a5ab409ea (127.6.204.194:39897), b79fbd9f226f4270a50e10dce97bafad (127.6.204.195:46407)
I20240502 08:04:53.936699 12682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "212a1cd8d6244f73b013888b748c3ee8" candidate_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b79fbd9f226f4270a50e10dce97bafad" is_pre_election: true
I20240502 08:04:53.937425 12682 raft_consensus.cc:2454] T 212a1cd8d6244f73b013888b748c3ee8 P b79fbd9f226f4270a50e10dce97bafad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e739706a3cb4269b73b3dc63e28ccfe in term 0.
I20240502 08:04:53.939203 12499 leader_election.cc:302] T 212a1cd8d6244f73b013888b748c3ee8 P 7e739706a3cb4269b73b3dc63e28ccfe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e739706a3cb4269b73b3dc63e28ccfe, b79fbd9f226f4270a50e10dce97bafad; no voters:
I20240502 08:04:53.941259 12780 raft_consensus.cc:2781] T 212a1cd8d6244f73b013888b748c3ee8 P 7e739706a3cb4269b73b3dc63e28ccfe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:04:53.941658 12780 raft_consensus.cc:482] T 212a1cd8d6244f73b013888b748c3ee8 P 7e739706a3cb4269b73b3dc63e28ccfe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:04:53.941991 12780 raft_consensus.cc:3037] T 212a1cd8d6244f73b013888b748c3ee8 P 7e739706a3cb4269b73b3dc63e28ccfe [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:53.939952 12429 catalog_manager.cc:5484] T bb44f0c8f9164047ae9d2ce1b67ab9da P 7e739706a3cb4269b73b3dc63e28ccfe reported cstate change: term changed from 0 to 1, leader changed from <none> to 7e739706a3cb4269b73b3dc63e28ccfe (127.6.204.193). New cstate: current_term: 1 leader_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } health_report { overall_health: UNKNOWN } } }
I20240502 08:04:53.946734 12608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "212a1cd8d6244f73b013888b748c3ee8" candidate_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca3300cf09c04d6cac79551a5ab409ea" is_pre_election: true
I20240502 08:04:53.954664 12683 raft_consensus.cc:2454] T 0f00e1259b934b7a904dbc7dc4a434ae P b79fbd9f226f4270a50e10dce97bafad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e739706a3cb4269b73b3dc63e28ccfe in term 1.
I20240502 08:04:53.963130 12786 raft_consensus.cc:482] T 9badc171df9a47f9bb33b775dea31604 P b79fbd9f226f4270a50e10dce97bafad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:04:53.962967 12731 consensus_queue.cc:1035] T 1abb47c8c0a642b097ae6327e60ec0e0 P 7e739706a3cb4269b73b3dc63e28ccfe [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:04:53.960430 12608 raft_consensus.cc:2454] T 212a1cd8d6244f73b013888b748c3ee8 P ca3300cf09c04d6cac79551a5ab409ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e739706a3cb4269b73b3dc63e28ccfe in term 0.
I20240502 08:04:53.965164 12786 raft_consensus.cc:504] T 9badc171df9a47f9bb33b775dea31604 P b79fbd9f226f4270a50e10dce97bafad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:53.967268 12780 raft_consensus.cc:504] T 212a1cd8d6244f73b013888b748c3ee8 P 7e739706a3cb4269b73b3dc63e28ccfe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:53.970842 12786 leader_election.cc:288] T 9badc171df9a47f9bb33b775dea31604 P b79fbd9f226f4270a50e10dce97bafad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e739706a3cb4269b73b3dc63e28ccfe (127.6.204.193:32777), ca3300cf09c04d6cac79551a5ab409ea (127.6.204.194:39897)
I20240502 08:04:53.974409 12608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "212a1cd8d6244f73b013888b748c3ee8" candidate_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca3300cf09c04d6cac79551a5ab409ea"
I20240502 08:04:53.975033 12608 raft_consensus.cc:3037] T 212a1cd8d6244f73b013888b748c3ee8 P ca3300cf09c04d6cac79551a5ab409ea [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:53.980062 12683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "212a1cd8d6244f73b013888b748c3ee8" candidate_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b79fbd9f226f4270a50e10dce97bafad"
I20240502 08:04:53.980158 12533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9badc171df9a47f9bb33b775dea31604" candidate_uuid: "b79fbd9f226f4270a50e10dce97bafad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" is_pre_election: true
I20240502 08:04:53.980779 12683 raft_consensus.cc:3037] T 212a1cd8d6244f73b013888b748c3ee8 P b79fbd9f226f4270a50e10dce97bafad [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:53.980852 12533 raft_consensus.cc:2454] T 9badc171df9a47f9bb33b775dea31604 P 7e739706a3cb4269b73b3dc63e28ccfe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b79fbd9f226f4270a50e10dce97bafad in term 0.
I20240502 08:04:53.983371 12787 consensus_queue.cc:1035] T 7fe7134eaf2e4e29ac762afe8855bdac P b79fbd9f226f4270a50e10dce97bafad [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 08:04:53.983496 12606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9badc171df9a47f9bb33b775dea31604" candidate_uuid: "b79fbd9f226f4270a50e10dce97bafad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca3300cf09c04d6cac79551a5ab409ea" is_pre_election: true
I20240502 08:04:53.984434 12606 raft_consensus.cc:2454] T 9badc171df9a47f9bb33b775dea31604 P ca3300cf09c04d6cac79551a5ab409ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b79fbd9f226f4270a50e10dce97bafad in term 0.
I20240502 08:04:53.988481 12683 raft_consensus.cc:2454] T 212a1cd8d6244f73b013888b748c3ee8 P b79fbd9f226f4270a50e10dce97bafad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e739706a3cb4269b73b3dc63e28ccfe in term 1.
I20240502 08:04:53.989576 12499 leader_election.cc:302] T 212a1cd8d6244f73b013888b748c3ee8 P 7e739706a3cb4269b73b3dc63e28ccfe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e739706a3cb4269b73b3dc63e28ccfe, b79fbd9f226f4270a50e10dce97bafad; no voters:
I20240502 08:04:53.990382 12780 leader_election.cc:288] T 212a1cd8d6244f73b013888b748c3ee8 P 7e739706a3cb4269b73b3dc63e28ccfe [CANDIDATE]: Term 1 election: Requested vote from peers ca3300cf09c04d6cac79551a5ab409ea (127.6.204.194:39897), b79fbd9f226f4270a50e10dce97bafad (127.6.204.195:46407)
I20240502 08:04:53.990900 12780 raft_consensus.cc:2781] T 212a1cd8d6244f73b013888b748c3ee8 P 7e739706a3cb4269b73b3dc63e28ccfe [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:04:53.991328 12780 raft_consensus.cc:686] T 212a1cd8d6244f73b013888b748c3ee8 P 7e739706a3cb4269b73b3dc63e28ccfe [term 1 LEADER]: Becoming Leader. State: Replica: 7e739706a3cb4269b73b3dc63e28ccfe, State: Running, Role: LEADER
I20240502 08:04:53.992136 12780 consensus_queue.cc:237] T 212a1cd8d6244f73b013888b748c3ee8 P 7e739706a3cb4269b73b3dc63e28ccfe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:53.994344 12787 consensus_queue.cc:1035] T bde9453a83b8464899af354b5e24806f P b79fbd9f226f4270a50e10dce97bafad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:04:53.996414 12647 leader_election.cc:302] T 9badc171df9a47f9bb33b775dea31604 P b79fbd9f226f4270a50e10dce97bafad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e739706a3cb4269b73b3dc63e28ccfe, b79fbd9f226f4270a50e10dce97bafad; no voters:
I20240502 08:04:53.997958 12787 raft_consensus.cc:2781] T 9badc171df9a47f9bb33b775dea31604 P b79fbd9f226f4270a50e10dce97bafad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:04:53.998348 12787 raft_consensus.cc:482] T 9badc171df9a47f9bb33b775dea31604 P b79fbd9f226f4270a50e10dce97bafad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:04:53.998669 12787 raft_consensus.cc:3037] T 9badc171df9a47f9bb33b775dea31604 P b79fbd9f226f4270a50e10dce97bafad [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:53.999269 12608 raft_consensus.cc:2454] T 212a1cd8d6244f73b013888b748c3ee8 P ca3300cf09c04d6cac79551a5ab409ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e739706a3cb4269b73b3dc63e28ccfe in term 1.
I20240502 08:04:53.999982 12431 catalog_manager.cc:5484] T 0f00e1259b934b7a904dbc7dc4a434ae P 7e739706a3cb4269b73b3dc63e28ccfe reported cstate change: term changed from 0 to 1, leader changed from <none> to 7e739706a3cb4269b73b3dc63e28ccfe (127.6.204.193). New cstate: current_term: 1 leader_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } health_report { overall_health: UNKNOWN } } }
I20240502 08:04:54.017550 12731 consensus_queue.cc:1035] T 1abb47c8c0a642b097ae6327e60ec0e0 P 7e739706a3cb4269b73b3dc63e28ccfe [LEADER]: Connected to new peer: Peer: permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 08:04:54.031828 12787 raft_consensus.cc:504] T 9badc171df9a47f9bb33b775dea31604 P b79fbd9f226f4270a50e10dce97bafad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:54.031324 12431 catalog_manager.cc:5484] T 212a1cd8d6244f73b013888b748c3ee8 P 7e739706a3cb4269b73b3dc63e28ccfe reported cstate change: term changed from 0 to 1, leader changed from <none> to 7e739706a3cb4269b73b3dc63e28ccfe (127.6.204.193). New cstate: current_term: 1 leader_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } health_report { overall_health: UNKNOWN } } }
I20240502 08:04:54.033504 12787 leader_election.cc:288] T 9badc171df9a47f9bb33b775dea31604 P b79fbd9f226f4270a50e10dce97bafad [CANDIDATE]: Term 1 election: Requested vote from peers 7e739706a3cb4269b73b3dc63e28ccfe (127.6.204.193:32777), ca3300cf09c04d6cac79551a5ab409ea (127.6.204.194:39897)
I20240502 08:04:54.034272 12534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9badc171df9a47f9bb33b775dea31604" candidate_uuid: "b79fbd9f226f4270a50e10dce97bafad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e739706a3cb4269b73b3dc63e28ccfe"
I20240502 08:04:54.034915 12534 raft_consensus.cc:3037] T 9badc171df9a47f9bb33b775dea31604 P 7e739706a3cb4269b73b3dc63e28ccfe [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:54.042032 12534 raft_consensus.cc:2454] T 9badc171df9a47f9bb33b775dea31604 P 7e739706a3cb4269b73b3dc63e28ccfe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b79fbd9f226f4270a50e10dce97bafad in term 1.
I20240502 08:04:54.045442 12609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9badc171df9a47f9bb33b775dea31604" candidate_uuid: "b79fbd9f226f4270a50e10dce97bafad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca3300cf09c04d6cac79551a5ab409ea"
I20240502 08:04:54.046070 12609 raft_consensus.cc:3037] T 9badc171df9a47f9bb33b775dea31604 P ca3300cf09c04d6cac79551a5ab409ea [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:04:54.053567 12609 raft_consensus.cc:2454] T 9badc171df9a47f9bb33b775dea31604 P ca3300cf09c04d6cac79551a5ab409ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b79fbd9f226f4270a50e10dce97bafad in term 1.
I20240502 08:04:54.053761 12735 consensus_queue.cc:1035] T 90b6d92c72f44d89a54c7b1b55470f4b P ca3300cf09c04d6cac79551a5ab409ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 08:04:54.054643 12647 leader_election.cc:302] T 9badc171df9a47f9bb33b775dea31604 P b79fbd9f226f4270a50e10dce97bafad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e739706a3cb4269b73b3dc63e28ccfe, b79fbd9f226f4270a50e10dce97bafad; no voters:
I20240502 08:04:54.055879 12785 raft_consensus.cc:2781] T 9badc171df9a47f9bb33b775dea31604 P b79fbd9f226f4270a50e10dce97bafad [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:04:54.056345 12785 raft_consensus.cc:686] T 9badc171df9a47f9bb33b775dea31604 P b79fbd9f226f4270a50e10dce97bafad [term 1 LEADER]: Becoming Leader. State: Replica: b79fbd9f226f4270a50e10dce97bafad, State: Running, Role: LEADER
I20240502 08:04:54.057137 12785 consensus_queue.cc:237] T 9badc171df9a47f9bb33b775dea31604 P b79fbd9f226f4270a50e10dce97bafad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:54.070655 12757 consensus_queue.cc:1035] T 90b6d92c72f44d89a54c7b1b55470f4b P ca3300cf09c04d6cac79551a5ab409ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 08:04:54.084906 12432 catalog_manager.cc:5484] T 9badc171df9a47f9bb33b775dea31604 P b79fbd9f226f4270a50e10dce97bafad reported cstate change: term changed from 0 to 1, leader changed from <none> to b79fbd9f226f4270a50e10dce97bafad (127.6.204.195). New cstate: current_term: 1 leader_uuid: "b79fbd9f226f4270a50e10dce97bafad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } health_report { overall_health: HEALTHY } } }
I20240502 08:04:54.171759 6963 tablet_server.cc:178] TabletServer@127.6.204.193:0 shutting down...
I20240502 08:04:54.190374 6963 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 08:04:54.191228 6963 tablet_replica.cc:330] T 212a1cd8d6244f73b013888b748c3ee8 P 7e739706a3cb4269b73b3dc63e28ccfe: stopping tablet replica
I20240502 08:04:54.191865 6963 raft_consensus.cc:2229] T 212a1cd8d6244f73b013888b748c3ee8 P 7e739706a3cb4269b73b3dc63e28ccfe [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:04:54.192422 6963 pending_rounds.cc:62] T 212a1cd8d6244f73b013888b748c3ee8 P 7e739706a3cb4269b73b3dc63e28ccfe: Trying to abort 1 pending ops.
I20240502 08:04:54.192584 6963 pending_rounds.cc:68] T 212a1cd8d6244f73b013888b748c3ee8 P 7e739706a3cb4269b73b3dc63e28ccfe: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023153537029038080 op_type: NO_OP noop_request { }
I20240502 08:04:54.192858 6963 raft_consensus.cc:2866] T 212a1cd8d6244f73b013888b748c3ee8 P 7e739706a3cb4269b73b3dc63e28ccfe [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240502 08:04:54.193080 6963 raft_consensus.cc:2258] T 212a1cd8d6244f73b013888b748c3ee8 P 7e739706a3cb4269b73b3dc63e28ccfe [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:04:54.195281 6963 tablet_replica.cc:330] T 9badc171df9a47f9bb33b775dea31604 P 7e739706a3cb4269b73b3dc63e28ccfe: stopping tablet replica
I20240502 08:04:54.195782 6963 raft_consensus.cc:2229] T 9badc171df9a47f9bb33b775dea31604 P 7e739706a3cb4269b73b3dc63e28ccfe [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:04:54.196099 6963 raft_consensus.cc:2258] T 9badc171df9a47f9bb33b775dea31604 P 7e739706a3cb4269b73b3dc63e28ccfe [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:04:54.197783 6963 tablet_replica.cc:330] T 0f00e1259b934b7a904dbc7dc4a434ae P 7e739706a3cb4269b73b3dc63e28ccfe: stopping tablet replica
I20240502 08:04:54.198310 6963 raft_consensus.cc:2229] T 0f00e1259b934b7a904dbc7dc4a434ae P 7e739706a3cb4269b73b3dc63e28ccfe [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:04:54.198802 6963 pending_rounds.cc:62] T 0f00e1259b934b7a904dbc7dc4a434ae P 7e739706a3cb4269b73b3dc63e28ccfe: Trying to abort 1 pending ops.
I20240502 08:04:54.198949 6963 pending_rounds.cc:68] T 0f00e1259b934b7a904dbc7dc4a434ae P 7e739706a3cb4269b73b3dc63e28ccfe: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023153536868663296 op_type: NO_OP noop_request { }
I20240502 08:04:54.199191 6963 raft_consensus.cc:2866] T 0f00e1259b934b7a904dbc7dc4a434ae P 7e739706a3cb4269b73b3dc63e28ccfe [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240502 08:04:54.199420 6963 raft_consensus.cc:2258] T 0f00e1259b934b7a904dbc7dc4a434ae P 7e739706a3cb4269b73b3dc63e28ccfe [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:04:54.201653 6963 tablet_replica.cc:330] T 90b6d92c72f44d89a54c7b1b55470f4b P 7e739706a3cb4269b73b3dc63e28ccfe: stopping tablet replica
I20240502 08:04:54.202208 6963 raft_consensus.cc:2229] T 90b6d92c72f44d89a54c7b1b55470f4b P 7e739706a3cb4269b73b3dc63e28ccfe [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:04:54.202572 6963 raft_consensus.cc:2258] T 90b6d92c72f44d89a54c7b1b55470f4b P 7e739706a3cb4269b73b3dc63e28ccfe [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:04:54.204607 6963 tablet_replica.cc:330] T 1abb47c8c0a642b097ae6327e60ec0e0 P 7e739706a3cb4269b73b3dc63e28ccfe: stopping tablet replica
I20240502 08:04:54.205068 6963 raft_consensus.cc:2229] T 1abb47c8c0a642b097ae6327e60ec0e0 P 7e739706a3cb4269b73b3dc63e28ccfe [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:04:54.205757 6963 raft_consensus.cc:2258] T 1abb47c8c0a642b097ae6327e60ec0e0 P 7e739706a3cb4269b73b3dc63e28ccfe [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:04:54.207573 6963 tablet_replica.cc:330] T 7fe7134eaf2e4e29ac762afe8855bdac P 7e739706a3cb4269b73b3dc63e28ccfe: stopping tablet replica
I20240502 08:04:54.208006 6963 raft_consensus.cc:2229] T 7fe7134eaf2e4e29ac762afe8855bdac P 7e739706a3cb4269b73b3dc63e28ccfe [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:04:54.208333 6963 raft_consensus.cc:2258] T 7fe7134eaf2e4e29ac762afe8855bdac P 7e739706a3cb4269b73b3dc63e28ccfe [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:04:54.210233 6963 tablet_replica.cc:330] T bb44f0c8f9164047ae9d2ce1b67ab9da P 7e739706a3cb4269b73b3dc63e28ccfe: stopping tablet replica
I20240502 08:04:54.210671 6963 raft_consensus.cc:2229] T bb44f0c8f9164047ae9d2ce1b67ab9da P 7e739706a3cb4269b73b3dc63e28ccfe [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:04:54.211151 6963 pending_rounds.cc:62] T bb44f0c8f9164047ae9d2ce1b67ab9da P 7e739706a3cb4269b73b3dc63e28ccfe: Trying to abort 1 pending ops.
I20240502 08:04:54.211302 6963 pending_rounds.cc:68] T bb44f0c8f9164047ae9d2ce1b67ab9da P 7e739706a3cb4269b73b3dc63e28ccfe: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023153536508051456 op_type: NO_OP noop_request { }
I20240502 08:04:54.211542 6963 raft_consensus.cc:2866] T bb44f0c8f9164047ae9d2ce1b67ab9da P 7e739706a3cb4269b73b3dc63e28ccfe [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240502 08:04:54.211748 6963 raft_consensus.cc:2258] T bb44f0c8f9164047ae9d2ce1b67ab9da P 7e739706a3cb4269b73b3dc63e28ccfe [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:04:54.213742 6963 tablet_replica.cc:330] T bde9453a83b8464899af354b5e24806f P 7e739706a3cb4269b73b3dc63e28ccfe: stopping tablet replica
I20240502 08:04:54.214202 6963 raft_consensus.cc:2229] T bde9453a83b8464899af354b5e24806f P 7e739706a3cb4269b73b3dc63e28ccfe [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:04:54.214531 6963 raft_consensus.cc:2258] T bde9453a83b8464899af354b5e24806f P 7e739706a3cb4269b73b3dc63e28ccfe [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:04:54.241922 6963 tablet_server.cc:194] TabletServer@127.6.204.193:0 shutdown complete.
W20240502 08:04:54.323855 12647 consensus_peers.cc:485] T 7fe7134eaf2e4e29ac762afe8855bdac P b79fbd9f226f4270a50e10dce97bafad -> Peer 7e739706a3cb4269b73b3dc63e28ccfe (127.6.204.193:32777): Couldn't send request to peer 7e739706a3cb4269b73b3dc63e28ccfe. Status: Network error: Client connection negotiation failed: client connection to 127.6.204.193:32777: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:04:54.459206 12647 consensus_peers.cc:485] T 9badc171df9a47f9bb33b775dea31604 P b79fbd9f226f4270a50e10dce97bafad -> Peer 7e739706a3cb4269b73b3dc63e28ccfe (127.6.204.193:32777): Couldn't send request to peer 7e739706a3cb4269b73b3dc63e28ccfe. Status: Network error: Client connection negotiation failed: client connection to 127.6.204.193:32777: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:04:54.579748 12647 consensus_peers.cc:485] T bde9453a83b8464899af354b5e24806f P b79fbd9f226f4270a50e10dce97bafad -> Peer 7e739706a3cb4269b73b3dc63e28ccfe (127.6.204.193:32777): Couldn't send request to peer 7e739706a3cb4269b73b3dc63e28ccfe. Status: Network error: Client connection negotiation failed: client connection to 127.6.204.193:32777: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:04:54.654980 12573 consensus_peers.cc:485] T 90b6d92c72f44d89a54c7b1b55470f4b P ca3300cf09c04d6cac79551a5ab409ea -> Peer 7e739706a3cb4269b73b3dc63e28ccfe (127.6.204.193:32777): Couldn't send request to peer 7e739706a3cb4269b73b3dc63e28ccfe. Status: Network error: Client connection negotiation failed: client connection to 127.6.204.193:32777: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:04:54.667045 12786 consensus_queue.cc:1035] T 9badc171df9a47f9bb33b775dea31604 P b79fbd9f226f4270a50e10dce97bafad [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 08:04:55.214591 12735 consensus_queue.cc:579] T 90b6d92c72f44d89a54c7b1b55470f4b P ca3300cf09c04d6cac79551a5ab409ea [LEADER]: Leader has been unable to successfully communicate with peer 7e739706a3cb4269b73b3dc63e28ccfe for more than 1 seconds (1.126s)
W20240502 08:04:55.217975 12573 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.204.193:32777: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 08:04:55.222476 12429 catalog_manager.cc:5162] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 90b6d92c72f44d89a54c7b1b55470f4b with cas_config_opid_index -1: no extra replica candidate found for tablet 90b6d92c72f44d89a54c7b1b55470f4b (table test-workload [id=08210cb430b440d0a7d1151c17012950]): Not found: could not select location for extra replica: not enough tablet servers to satisfy replica placement policy: the total number of registered tablet servers (3) does not allow for adding an extra replica; consider bringing up more to have at least 4 tablet servers up and running
I20240502 08:04:55.268124 12786 consensus_queue.cc:579] T 7fe7134eaf2e4e29ac762afe8855bdac P b79fbd9f226f4270a50e10dce97bafad [LEADER]: Leader has been unable to successfully communicate with peer 7e739706a3cb4269b73b3dc63e28ccfe for more than 1 seconds (1.298s)
I20240502 08:04:55.275915 12429 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47756:
table { table_name: "dugtrio" } modify_external_catalogs: true
I20240502 08:04:55.276801 12429 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47756:
table { table_name: "dugtrio" } modify_external_catalogs: true
I20240502 08:04:55.296712 12429 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b4c502a6b1fa4eb79c635850e5fa3883: Sending DeleteTablet for 3 replicas of tablet 0f00e1259b934b7a904dbc7dc4a434ae
I20240502 08:04:55.298702 12429 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b4c502a6b1fa4eb79c635850e5fa3883: Sending DeleteTablet for 3 replicas of tablet bb44f0c8f9164047ae9d2ce1b67ab9da
I20240502 08:04:55.300069 12672 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f00e1259b934b7a904dbc7dc4a434ae with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:04:55 UTC) from {username='slave'} at 127.0.0.1:46014
I20240502 08:04:55.300427 12429 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b4c502a6b1fa4eb79c635850e5fa3883: Sending DeleteTablet for 3 replicas of tablet 9badc171df9a47f9bb33b775dea31604
I20240502 08:04:55.302155 12429 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b4c502a6b1fa4eb79c635850e5fa3883: Sending DeleteTablet for 3 replicas of tablet 212a1cd8d6244f73b013888b748c3ee8
I20240502 08:04:55.302338 12597 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f00e1259b934b7a904dbc7dc4a434ae with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:04:55 UTC) from {username='slave'} at 127.0.0.1:60006
I20240502 08:04:55.302377 12671 tablet_service.cc:1514] Processing DeleteTablet for tablet bb44f0c8f9164047ae9d2ce1b67ab9da with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:04:55 UTC) from {username='slave'} at 127.0.0.1:46014
I20240502 08:04:55.309384 6963 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 08:04:55.305915 12599 tablet_service.cc:1514] Processing DeleteTablet for tablet bb44f0c8f9164047ae9d2ce1b67ab9da with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:04:55 UTC) from {username='slave'} at 127.0.0.1:60006
I20240502 08:04:55.311033 12671 tablet_service.cc:1514] Processing DeleteTablet for tablet 9badc171df9a47f9bb33b775dea31604 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:04:55 UTC) from {username='slave'} at 127.0.0.1:46014
I20240502 08:04:55.313499 12671 tablet_service.cc:1514] Processing DeleteTablet for tablet 212a1cd8d6244f73b013888b748c3ee8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:04:55 UTC) from {username='slave'} at 127.0.0.1:46014
I20240502 08:04:55.314363 12824 tablet_replica.cc:330] T 0f00e1259b934b7a904dbc7dc4a434ae P ca3300cf09c04d6cac79551a5ab409ea: stopping tablet replica
I20240502 08:04:55.316329 12824 raft_consensus.cc:2229] T 0f00e1259b934b7a904dbc7dc4a434ae P ca3300cf09c04d6cac79551a5ab409ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:04:55.317291 12824 raft_consensus.cc:2258] T 0f00e1259b934b7a904dbc7dc4a434ae P ca3300cf09c04d6cac79551a5ab409ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:04:55.317695 12823 tablet_replica.cc:330] T 0f00e1259b934b7a904dbc7dc4a434ae P b79fbd9f226f4270a50e10dce97bafad: stopping tablet replica
I20240502 08:04:55.319106 12823 raft_consensus.cc:2229] T 0f00e1259b934b7a904dbc7dc4a434ae P b79fbd9f226f4270a50e10dce97bafad [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:04:55.320036 12823 raft_consensus.cc:2258] T 0f00e1259b934b7a904dbc7dc4a434ae P b79fbd9f226f4270a50e10dce97bafad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:04:55.320084 12597 tablet_service.cc:1514] Processing DeleteTablet for tablet 9badc171df9a47f9bb33b775dea31604 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:04:55 UTC) from {username='slave'} at 127.0.0.1:60006
I20240502 08:04:55.321419 12824 ts_tablet_manager.cc:1875] T 0f00e1259b934b7a904dbc7dc4a434ae P ca3300cf09c04d6cac79551a5ab409ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:04:55.322906 12597 tablet_service.cc:1514] Processing DeleteTablet for tablet 212a1cd8d6244f73b013888b748c3ee8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:04:55 UTC) from {username='slave'} at 127.0.0.1:60006
I20240502 08:04:55.323902 12823 ts_tablet_manager.cc:1875] T 0f00e1259b934b7a904dbc7dc4a434ae P b79fbd9f226f4270a50e10dce97bafad: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 08:04:55.324719 12417 catalog_manager.cc:4573] TS 7e739706a3cb4269b73b3dc63e28ccfe (127.6.204.193:32777): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0f00e1259b934b7a904dbc7dc4a434ae: Network error: Client connection negotiation failed: client connection to 127.6.204.193:32777: connect: Connection refused (error 111)
W20240502 08:04:55.337723 12826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:04:55.339336 12827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:04:55.345721 12829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:04:55.346494 12824 ts_tablet_manager.cc:1888] T 0f00e1259b934b7a904dbc7dc4a434ae P ca3300cf09c04d6cac79551a5ab409ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:04:55.347030 12824 log.cc:1196] T 0f00e1259b934b7a904dbc7dc4a434ae P ca3300cf09c04d6cac79551a5ab409ea: Deleting WAL directory at /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714636933625983-6963-0/minicluster-data/ts-1-root/wals/0f00e1259b934b7a904dbc7dc4a434ae
I20240502 08:04:55.347040 6963 server_base.cc:1034] running on GCE node
I20240502 08:04:55.347501 12823 ts_tablet_manager.cc:1888] T 0f00e1259b934b7a904dbc7dc4a434ae P b79fbd9f226f4270a50e10dce97bafad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:04:55.347934 12823 log.cc:1196] T 0f00e1259b934b7a904dbc7dc4a434ae P b79fbd9f226f4270a50e10dce97bafad: Deleting WAL directory at /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714636933625983-6963-0/minicluster-data/ts-2-root/wals/0f00e1259b934b7a904dbc7dc4a434ae
I20240502 08:04:55.348121 12824 ts_tablet_manager.cc:1909] T 0f00e1259b934b7a904dbc7dc4a434ae P ca3300cf09c04d6cac79551a5ab409ea: Deleting consensus metadata
I20240502 08:04:55.348506 6963 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20240502 08:04:55.348780 12823 ts_tablet_manager.cc:1909] T 0f00e1259b934b7a904dbc7dc4a434ae P b79fbd9f226f4270a50e10dce97bafad: Deleting consensus metadata
W20240502 08:04:55.348831 6963 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:04:55.349231 6963 hybrid_clock.cc:648] HybridClock initialized: now 1714637095349211 us; error 0 us; skew 500 ppm
I20240502 08:04:55.350304 6963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:04:55.352649 12824 tablet_replica.cc:330] T bb44f0c8f9164047ae9d2ce1b67ab9da P ca3300cf09c04d6cac79551a5ab409ea: stopping tablet replica
I20240502 08:04:55.353564 12418 catalog_manager.cc:4830] TS b79fbd9f226f4270a50e10dce97bafad (127.6.204.195:46407): tablet 0f00e1259b934b7a904dbc7dc4a434ae (table dugtrio [id=d58d27d9f8724a5ca12188de828572b7]) successfully deleted
I20240502 08:04:55.353976 12417 catalog_manager.cc:4830] TS ca3300cf09c04d6cac79551a5ab409ea (127.6.204.194:39897): tablet 0f00e1259b934b7a904dbc7dc4a434ae (table dugtrio [id=d58d27d9f8724a5ca12188de828572b7]) successfully deleted
I20240502 08:04:55.353850 12824 raft_consensus.cc:2229] T bb44f0c8f9164047ae9d2ce1b67ab9da P ca3300cf09c04d6cac79551a5ab409ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:04:55.354983 12823 tablet_replica.cc:330] T bb44f0c8f9164047ae9d2ce1b67ab9da P b79fbd9f226f4270a50e10dce97bafad: stopping tablet replica
I20240502 08:04:55.355315 12824 raft_consensus.cc:2258] T bb44f0c8f9164047ae9d2ce1b67ab9da P ca3300cf09c04d6cac79551a5ab409ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:04:55.356324 12823 raft_consensus.cc:2229] T bb44f0c8f9164047ae9d2ce1b67ab9da P b79fbd9f226f4270a50e10dce97bafad [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:04:55.357000 12823 raft_consensus.cc:2258] T bb44f0c8f9164047ae9d2ce1b67ab9da P b79fbd9f226f4270a50e10dce97bafad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:04:55.359598 12824 ts_tablet_manager.cc:1875] T bb44f0c8f9164047ae9d2ce1b67ab9da P ca3300cf09c04d6cac79551a5ab409ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:04:55.359692 6963 webserver.cc:460] Webserver started at http://127.6.204.196:39925/ using document root <none> and password file <none>
I20240502 08:04:55.360559 6963 fs_manager.cc:359] Metadata directory not provided
I20240502 08:04:55.360848 6963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:04:55.360997 12823 ts_tablet_manager.cc:1875] T bb44f0c8f9164047ae9d2ce1b67ab9da P b79fbd9f226f4270a50e10dce97bafad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:04:55.361244 6963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:04:55.363075 6963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714636933625983-6963-0/minicluster-data/ts-3-root/instance:
uuid: "701bb8dd0643469ab2ead20bab8b0451"
format_stamp: "Formatted at 2024-05-02 08:04:55 on dist-test-slave-32xs"
I20240502 08:04:55.369921 6963 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20240502 08:04:55.371686 12824 ts_tablet_manager.cc:1888] T bb44f0c8f9164047ae9d2ce1b67ab9da P ca3300cf09c04d6cac79551a5ab409ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:04:55.372042 12824 log.cc:1196] T bb44f0c8f9164047ae9d2ce1b67ab9da P ca3300cf09c04d6cac79551a5ab409ea: Deleting WAL directory at /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714636933625983-6963-0/minicluster-data/ts-1-root/wals/bb44f0c8f9164047ae9d2ce1b67ab9da
I20240502 08:04:55.372586 12824 ts_tablet_manager.cc:1909] T bb44f0c8f9164047ae9d2ce1b67ab9da P ca3300cf09c04d6cac79551a5ab409ea: Deleting consensus metadata
I20240502 08:04:55.375303 12834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:04:55.375912 12824 tablet_replica.cc:330] T 9badc171df9a47f9bb33b775dea31604 P ca3300cf09c04d6cac79551a5ab409ea: stopping tablet replica
I20240502 08:04:55.376022 12417 catalog_manager.cc:4830] TS ca3300cf09c04d6cac79551a5ab409ea (127.6.204.194:39897): tablet bb44f0c8f9164047ae9d2ce1b67ab9da (table dugtrio [id=d58d27d9f8724a5ca12188de828572b7]) successfully deleted
I20240502 08:04:55.376529 12823 ts_tablet_manager.cc:1888] T bb44f0c8f9164047ae9d2ce1b67ab9da P b79fbd9f226f4270a50e10dce97bafad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:04:55.376724 6963 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20240502 08:04:55.376973 12823 log.cc:1196] T bb44f0c8f9164047ae9d2ce1b67ab9da P b79fbd9f226f4270a50e10dce97bafad: Deleting WAL directory at /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714636933625983-6963-0/minicluster-data/ts-2-root/wals/bb44f0c8f9164047ae9d2ce1b67ab9da
I20240502 08:04:55.377102 12824 raft_consensus.cc:2229] T 9badc171df9a47f9bb33b775dea31604 P ca3300cf09c04d6cac79551a5ab409ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:04:55.377112 6963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714636933625983-6963-0/minicluster-data/ts-3-root
uuid: "701bb8dd0643469ab2ead20bab8b0451"
format_stamp: "Formatted at 2024-05-02 08:04:55 on dist-test-slave-32xs"
I20240502 08:04:55.377725 12823 ts_tablet_manager.cc:1909] T bb44f0c8f9164047ae9d2ce1b67ab9da P b79fbd9f226f4270a50e10dce97bafad: Deleting consensus metadata
I20240502 08:04:55.377780 12824 raft_consensus.cc:2258] T 9badc171df9a47f9bb33b775dea31604 P ca3300cf09c04d6cac79551a5ab409ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:04:55.377751 6963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714636933625983-6963-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714636933625983-6963-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714636933625983-6963-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:04:55.381273 12824 ts_tablet_manager.cc:1875] T 9badc171df9a47f9bb33b775dea31604 P ca3300cf09c04d6cac79551a5ab409ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:04:55.381596 12418 catalog_manager.cc:4830] TS b79fbd9f226f4270a50e10dce97bafad (127.6.204.195:46407): tablet bb44f0c8f9164047ae9d2ce1b67ab9da (table dugtrio [id=d58d27d9f8724a5ca12188de828572b7]) successfully deleted
I20240502 08:04:55.382907 12823 tablet_replica.cc:330] T 9badc171df9a47f9bb33b775dea31604 P b79fbd9f226f4270a50e10dce97bafad: stopping tablet replica
I20240502 08:04:55.384227 12823 raft_consensus.cc:2229] T 9badc171df9a47f9bb33b775dea31604 P b79fbd9f226f4270a50e10dce97bafad [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:04:55.385681 12823 raft_consensus.cc:2258] T 9badc171df9a47f9bb33b775dea31604 P b79fbd9f226f4270a50e10dce97bafad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:04:55.390462 12823 ts_tablet_manager.cc:1875] T 9badc171df9a47f9bb33b775dea31604 P b79fbd9f226f4270a50e10dce97bafad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:04:55.410372 12824 ts_tablet_manager.cc:1888] T 9badc171df9a47f9bb33b775dea31604 P ca3300cf09c04d6cac79551a5ab409ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:04:55.410807 12824 log.cc:1196] T 9badc171df9a47f9bb33b775dea31604 P ca3300cf09c04d6cac79551a5ab409ea: Deleting WAL directory at /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714636933625983-6963-0/minicluster-data/ts-1-root/wals/9badc171df9a47f9bb33b775dea31604
I20240502 08:04:55.411422 12824 ts_tablet_manager.cc:1909] T 9badc171df9a47f9bb33b775dea31604 P ca3300cf09c04d6cac79551a5ab409ea: Deleting consensus metadata
I20240502 08:04:55.414249 12824 tablet_replica.cc:330] T 212a1cd8d6244f73b013888b748c3ee8 P ca3300cf09c04d6cac79551a5ab409ea: stopping tablet replica
I20240502 08:04:55.414458 12417 catalog_manager.cc:4830] TS ca3300cf09c04d6cac79551a5ab409ea (127.6.204.194:39897): tablet 9badc171df9a47f9bb33b775dea31604 (table dugtrio [id=d58d27d9f8724a5ca12188de828572b7]) successfully deleted
I20240502 08:04:55.415170 12824 raft_consensus.cc:2229] T 212a1cd8d6244f73b013888b748c3ee8 P ca3300cf09c04d6cac79551a5ab409ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:04:55.415766 12824 raft_consensus.cc:2258] T 212a1cd8d6244f73b013888b748c3ee8 P ca3300cf09c04d6cac79551a5ab409ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:04:55.418273 12824 ts_tablet_manager.cc:1875] T 212a1cd8d6244f73b013888b748c3ee8 P ca3300cf09c04d6cac79551a5ab409ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:04:55.419730 12823 ts_tablet_manager.cc:1888] T 9badc171df9a47f9bb33b775dea31604 P b79fbd9f226f4270a50e10dce97bafad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:04:55.420132 12823 log.cc:1196] T 9badc171df9a47f9bb33b775dea31604 P b79fbd9f226f4270a50e10dce97bafad: Deleting WAL directory at /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714636933625983-6963-0/minicluster-data/ts-2-root/wals/9badc171df9a47f9bb33b775dea31604
I20240502 08:04:55.420868 12823 ts_tablet_manager.cc:1909] T 9badc171df9a47f9bb33b775dea31604 P b79fbd9f226f4270a50e10dce97bafad: Deleting consensus metadata
I20240502 08:04:55.424197 12418 catalog_manager.cc:4830] TS b79fbd9f226f4270a50e10dce97bafad (127.6.204.195:46407): tablet 9badc171df9a47f9bb33b775dea31604 (table dugtrio [id=d58d27d9f8724a5ca12188de828572b7]) successfully deleted
I20240502 08:04:55.425109 12823 tablet_replica.cc:330] T 212a1cd8d6244f73b013888b748c3ee8 P b79fbd9f226f4270a50e10dce97bafad: stopping tablet replica
I20240502 08:04:55.426205 12823 raft_consensus.cc:2229] T 212a1cd8d6244f73b013888b748c3ee8 P b79fbd9f226f4270a50e10dce97bafad [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:04:55.426826 12823 raft_consensus.cc:2258] T 212a1cd8d6244f73b013888b748c3ee8 P b79fbd9f226f4270a50e10dce97bafad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:04:55.429970 12823 ts_tablet_manager.cc:1875] T 212a1cd8d6244f73b013888b748c3ee8 P b79fbd9f226f4270a50e10dce97bafad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:04:55.430274 6963 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:04:55.432188 6963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:04:55.434114 12824 ts_tablet_manager.cc:1888] T 212a1cd8d6244f73b013888b748c3ee8 P ca3300cf09c04d6cac79551a5ab409ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:04:55.434612 12824 log.cc:1196] T 212a1cd8d6244f73b013888b748c3ee8 P ca3300cf09c04d6cac79551a5ab409ea: Deleting WAL directory at /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714636933625983-6963-0/minicluster-data/ts-1-root/wals/212a1cd8d6244f73b013888b748c3ee8
I20240502 08:04:55.434722 6963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:04:55.435379 12824 ts_tablet_manager.cc:1909] T 212a1cd8d6244f73b013888b748c3ee8 P ca3300cf09c04d6cac79551a5ab409ea: Deleting consensus metadata
I20240502 08:04:55.438246 12417 catalog_manager.cc:4830] TS ca3300cf09c04d6cac79551a5ab409ea (127.6.204.194:39897): tablet 212a1cd8d6244f73b013888b748c3ee8 (table dugtrio [id=d58d27d9f8724a5ca12188de828572b7]) successfully deleted
I20240502 08:04:55.438853 6963 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:04:55.439172 6963 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:04:55.439520 6963 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:04:55.439764 6963 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:04:55.449807 12823 ts_tablet_manager.cc:1888] T 212a1cd8d6244f73b013888b748c3ee8 P b79fbd9f226f4270a50e10dce97bafad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:04:55.450505 12823 log.cc:1196] T 212a1cd8d6244f73b013888b748c3ee8 P b79fbd9f226f4270a50e10dce97bafad: Deleting WAL directory at /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714636933625983-6963-0/minicluster-data/ts-2-root/wals/212a1cd8d6244f73b013888b748c3ee8
I20240502 08:04:55.451455 12823 ts_tablet_manager.cc:1909] T 212a1cd8d6244f73b013888b748c3ee8 P b79fbd9f226f4270a50e10dce97bafad: Deleting consensus metadata
I20240502 08:04:55.455489 12418 catalog_manager.cc:4830] TS b79fbd9f226f4270a50e10dce97bafad (127.6.204.195:46407): tablet 212a1cd8d6244f73b013888b748c3ee8 (table dugtrio [id=d58d27d9f8724a5ca12188de828572b7]) successfully deleted
I20240502 08:04:55.478983 12786 consensus_queue.cc:579] T bde9453a83b8464899af354b5e24806f P b79fbd9f226f4270a50e10dce97bafad [LEADER]: Leader has been unable to successfully communicate with peer 7e739706a3cb4269b73b3dc63e28ccfe for more than 1 seconds (1.418s)
I20240502 08:04:55.492942 6963 rpc_server.cc:307] RPC server started. Bound to: 127.6.204.196:37157
I20240502 08:04:55.493078 12896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.204.196:37157 every 8 connection(s)
I20240502 08:04:55.501195 6963 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:04:55.511586 12900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:04:55.513317 12901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:04:55.515089 12897 heartbeater.cc:347] Connected to a master server at 127.6.204.254:32811
I20240502 08:04:55.515825 12897 heartbeater.cc:464] Registering TS with master...
I20240502 08:04:55.516989 12897 heartbeater.cc:511] Master 127.6.204.254:32811 requested a full tablet report, sending...
W20240502 08:04:55.517275 12903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:04:55.514977 12813 raft_consensus.cc:482] T 1abb47c8c0a642b097ae6327e60ec0e0 P ca3300cf09c04d6cac79551a5ab409ea [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7e739706a3cb4269b73b3dc63e28ccfe)
I20240502 08:04:55.518493 12813 raft_consensus.cc:504] T 1abb47c8c0a642b097ae6327e60ec0e0 P ca3300cf09c04d6cac79551a5ab409ea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:55.519522 12429 ts_manager.cc:194] Registered new tserver with Master: 701bb8dd0643469ab2ead20bab8b0451 (127.6.204.196:37157)
I20240502 08:04:55.521153 12429 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47792
I20240502 08:04:55.521617 12786 raft_consensus.cc:482] T 1abb47c8c0a642b097ae6327e60ec0e0 P b79fbd9f226f4270a50e10dce97bafad [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7e739706a3cb4269b73b3dc63e28ccfe)
I20240502 08:04:55.522274 12786 raft_consensus.cc:504] T 1abb47c8c0a642b097ae6327e60ec0e0 P b79fbd9f226f4270a50e10dce97bafad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:55.524235 12786 leader_election.cc:288] T 1abb47c8c0a642b097ae6327e60ec0e0 P b79fbd9f226f4270a50e10dce97bafad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ca3300cf09c04d6cac79551a5ab409ea (127.6.204.194:39897), 7e739706a3cb4269b73b3dc63e28ccfe (127.6.204.193:32777)
I20240502 08:04:55.526229 12609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1abb47c8c0a642b097ae6327e60ec0e0" candidate_uuid: "b79fbd9f226f4270a50e10dce97bafad" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca3300cf09c04d6cac79551a5ab409ea" is_pre_election: true
I20240502 08:04:55.527278 12609 raft_consensus.cc:2454] T 1abb47c8c0a642b097ae6327e60ec0e0 P ca3300cf09c04d6cac79551a5ab409ea [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b79fbd9f226f4270a50e10dce97bafad in term 1.
W20240502 08:04:55.530229 12573 leader_election.cc:334] T 1abb47c8c0a642b097ae6327e60ec0e0 P ca3300cf09c04d6cac79551a5ab409ea [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7e739706a3cb4269b73b3dc63e28ccfe (127.6.204.193:32777): Network error: Client connection negotiation failed: client connection to 127.6.204.193:32777: connect: Connection refused (error 111)
W20240502 08:04:55.531227 12647 leader_election.cc:334] T 1abb47c8c0a642b097ae6327e60ec0e0 P b79fbd9f226f4270a50e10dce97bafad [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7e739706a3cb4269b73b3dc63e28ccfe (127.6.204.193:32777): Network error: Client connection negotiation failed: client connection to 127.6.204.193:32777: connect: Connection refused (error 111)
I20240502 08:04:55.532294 12647 leader_election.cc:302] T 1abb47c8c0a642b097ae6327e60ec0e0 P b79fbd9f226f4270a50e10dce97bafad [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b79fbd9f226f4270a50e10dce97bafad, ca3300cf09c04d6cac79551a5ab409ea; no voters: 7e739706a3cb4269b73b3dc63e28ccfe
I20240502 08:04:55.533360 12786 raft_consensus.cc:2781] T 1abb47c8c0a642b097ae6327e60ec0e0 P b79fbd9f226f4270a50e10dce97bafad [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:04:55.533695 12786 raft_consensus.cc:482] T 1abb47c8c0a642b097ae6327e60ec0e0 P b79fbd9f226f4270a50e10dce97bafad [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7e739706a3cb4269b73b3dc63e28ccfe)
I20240502 08:04:55.533999 12786 raft_consensus.cc:3037] T 1abb47c8c0a642b097ae6327e60ec0e0 P b79fbd9f226f4270a50e10dce97bafad [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:04:55.535621 12813 leader_election.cc:288] T 1abb47c8c0a642b097ae6327e60ec0e0 P ca3300cf09c04d6cac79551a5ab409ea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7e739706a3cb4269b73b3dc63e28ccfe (127.6.204.193:32777), b79fbd9f226f4270a50e10dce97bafad (127.6.204.195:46407)
I20240502 08:04:55.536160 12682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1abb47c8c0a642b097ae6327e60ec0e0" candidate_uuid: "ca3300cf09c04d6cac79551a5ab409ea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b79fbd9f226f4270a50e10dce97bafad" is_pre_election: true
I20240502 08:04:55.539824 6963 server_base.cc:1034] running on GCE node
I20240502 08:04:55.541035 6963 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 08:04:55.541293 6963 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 08:04:55.541512 6963 hybrid_clock.cc:648] HybridClock initialized: now 1714637095541488 us; error 0 us; skew 500 ppm
I20240502 08:04:55.542320 6963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:04:55.541863 12786 raft_consensus.cc:504] T 1abb47c8c0a642b097ae6327e60ec0e0 P b79fbd9f226f4270a50e10dce97bafad [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:55.543201 12682 raft_consensus.cc:2379] T 1abb47c8c0a642b097ae6327e60ec0e0 P b79fbd9f226f4270a50e10dce97bafad [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca3300cf09c04d6cac79551a5ab409ea in current term 2: Already voted for candidate b79fbd9f226f4270a50e10dce97bafad in this term.
I20240502 08:04:55.544044 12786 leader_election.cc:288] T 1abb47c8c0a642b097ae6327e60ec0e0 P b79fbd9f226f4270a50e10dce97bafad [CANDIDATE]: Term 2 election: Requested vote from peers ca3300cf09c04d6cac79551a5ab409ea (127.6.204.194:39897), 7e739706a3cb4269b73b3dc63e28ccfe (127.6.204.193:32777)
I20240502 08:04:55.542744 12683 consensus_queue.cc:237] T bde9453a83b8464899af354b5e24806f P b79fbd9f226f4270a50e10dce97bafad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } } peers { permanent_uuid: "701bb8dd0643469ab2ead20bab8b0451" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 37157 } attrs { promote: true } }
I20240502 08:04:55.546852 12574 leader_election.cc:302] T 1abb47c8c0a642b097ae6327e60ec0e0 P ca3300cf09c04d6cac79551a5ab409ea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca3300cf09c04d6cac79551a5ab409ea; no voters: 7e739706a3cb4269b73b3dc63e28ccfe, b79fbd9f226f4270a50e10dce97bafad
I20240502 08:04:55.547837 12813 raft_consensus.cc:3037] T 1abb47c8c0a642b097ae6327e60ec0e0 P ca3300cf09c04d6cac79551a5ab409ea [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:04:55.547886 12609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1abb47c8c0a642b097ae6327e60ec0e0" candidate_uuid: "b79fbd9f226f4270a50e10dce97bafad" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca3300cf09c04d6cac79551a5ab409ea"
W20240502 08:04:55.552232 12647 leader_election.cc:334] T 1abb47c8c0a642b097ae6327e60ec0e0 P b79fbd9f226f4270a50e10dce97bafad [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 7e739706a3cb4269b73b3dc63e28ccfe (127.6.204.193:32777): Network error: Client connection negotiation failed: client connection to 127.6.204.193:32777: connect: Connection refused (error 111)
I20240502 08:04:55.560639 12813 raft_consensus.cc:2726] T 1abb47c8c0a642b097ae6327e60ec0e0 P ca3300cf09c04d6cac79551a5ab409ea [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:04:55.561014 12606 raft_consensus.cc:1261] T bde9453a83b8464899af354b5e24806f P ca3300cf09c04d6cac79551a5ab409ea [term 1 FOLLOWER]: Refusing update from remote peer b79fbd9f226f4270a50e10dce97bafad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:04:55.563067 6963 webserver.cc:460] Webserver started at http://127.6.204.193:37837/ using document root <none> and password file <none>
I20240502 08:04:55.563663 6963 fs_manager.cc:359] Metadata directory not provided
I20240502 08:04:55.563905 6963 fs_manager.cc:362] Using existing metadata directory in first data directory
I20240502 08:04:55.567493 12786 consensus_queue.cc:1035] T bde9453a83b8464899af354b5e24806f P b79fbd9f226f4270a50e10dce97bafad [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:04:55.570102 6963 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.001s
I20240502 08:04:55.570087 12609 raft_consensus.cc:2454] T 1abb47c8c0a642b097ae6327e60ec0e0 P ca3300cf09c04d6cac79551a5ab409ea [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b79fbd9f226f4270a50e10dce97bafad in term 2.
W20240502 08:04:55.574561 12647 consensus_peers.cc:485] T bde9453a83b8464899af354b5e24806f P b79fbd9f226f4270a50e10dce97bafad -> Peer 7e739706a3cb4269b73b3dc63e28ccfe (127.6.204.193:32777): Couldn't send request to peer 7e739706a3cb4269b73b3dc63e28ccfe. Status: Network error: Client connection negotiation failed: client connection to 127.6.204.193:32777: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:04:55.575340 12647 leader_election.cc:302] T 1abb47c8c0a642b097ae6327e60ec0e0 P b79fbd9f226f4270a50e10dce97bafad [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b79fbd9f226f4270a50e10dce97bafad, ca3300cf09c04d6cac79551a5ab409ea; no voters: 7e739706a3cb4269b73b3dc63e28ccfe
I20240502 08:04:55.576383 12786 raft_consensus.cc:2781] T 1abb47c8c0a642b097ae6327e60ec0e0 P b79fbd9f226f4270a50e10dce97bafad [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:04:55.577406 12912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:04:55.577807 12786 raft_consensus.cc:686] T 1abb47c8c0a642b097ae6327e60ec0e0 P b79fbd9f226f4270a50e10dce97bafad [term 2 LEADER]: Becoming Leader. State: Replica: b79fbd9f226f4270a50e10dce97bafad, State: Running, Role: LEADER
I20240502 08:04:55.579036 12786 consensus_queue.cc:237] T 1abb47c8c0a642b097ae6327e60ec0e0 P b79fbd9f226f4270a50e10dce97bafad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:55.579775 6963 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20240502 08:04:55.580358 6963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714636933625983-6963-0/minicluster-data/ts-0-root
uuid: "7e739706a3cb4269b73b3dc63e28ccfe"
format_stamp: "Formatted at 2024-05-02 08:04:51 on dist-test-slave-32xs"
I20240502 08:04:55.580753 6963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714636933625983-6963-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714636933625983-6963-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714636933625983-6963-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:04:55.582388 12787 raft_consensus.cc:2932] T bde9453a83b8464899af354b5e24806f P b79fbd9f226f4270a50e10dce97bafad [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 701bb8dd0643469ab2ead20bab8b0451 (127.6.204.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } } peers { permanent_uuid: "701bb8dd0643469ab2ead20bab8b0451" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 37157 } attrs { promote: true } } }
W20240502 08:04:55.586709 12649 consensus_peers.cc:485] T bde9453a83b8464899af354b5e24806f P b79fbd9f226f4270a50e10dce97bafad -> Peer 701bb8dd0643469ab2ead20bab8b0451 (127.6.204.196:37157): Couldn't send request to peer 701bb8dd0643469ab2ead20bab8b0451. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bde9453a83b8464899af354b5e24806f. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:04:55.587658 12609 raft_consensus.cc:2932] T bde9453a83b8464899af354b5e24806f P ca3300cf09c04d6cac79551a5ab409ea [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 701bb8dd0643469ab2ead20bab8b0451 (127.6.204.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } } peers { permanent_uuid: "701bb8dd0643469ab2ead20bab8b0451" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 37157 } attrs { promote: true } } }
I20240502 08:04:55.595913 12683 consensus_queue.cc:237] T 7fe7134eaf2e4e29ac762afe8855bdac P b79fbd9f226f4270a50e10dce97bafad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } } peers { permanent_uuid: "701bb8dd0643469ab2ead20bab8b0451" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 37157 } attrs { promote: true } }
I20240502 08:04:55.613173 12418 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet bde9453a83b8464899af354b5e24806f with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 2)
I20240502 08:04:55.611590 12429 catalog_manager.cc:5484] T bde9453a83b8464899af354b5e24806f P ca3300cf09c04d6cac79551a5ab409ea reported cstate change: config changed from index -1 to 2, NON_VOTER 701bb8dd0643469ab2ead20bab8b0451 (127.6.204.196) added. New cstate: current_term: 1 leader_uuid: "b79fbd9f226f4270a50e10dce97bafad" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } } peers { permanent_uuid: "701bb8dd0643469ab2ead20bab8b0451" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 37157 } attrs { promote: true } } }
I20240502 08:04:55.618893 12609 consensus_queue.cc:237] T 90b6d92c72f44d89a54c7b1b55470f4b P ca3300cf09c04d6cac79551a5ab409ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } } peers { permanent_uuid: "701bb8dd0643469ab2ead20bab8b0451" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 37157 } attrs { promote: true } }
I20240502 08:04:55.620433 12430 catalog_manager.cc:5484] T 1abb47c8c0a642b097ae6327e60ec0e0 P b79fbd9f226f4270a50e10dce97bafad reported cstate change: term changed from 1 to 2, leader changed from 7e739706a3cb4269b73b3dc63e28ccfe (127.6.204.193) to b79fbd9f226f4270a50e10dce97bafad (127.6.204.195). New cstate: current_term: 2 leader_uuid: "b79fbd9f226f4270a50e10dce97bafad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } health_report { overall_health: HEALTHY } } }
I20240502 08:04:55.629223 12606 raft_consensus.cc:1261] T 7fe7134eaf2e4e29ac762afe8855bdac P ca3300cf09c04d6cac79551a5ab409ea [term 1 FOLLOWER]: Refusing update from remote peer b79fbd9f226f4270a50e10dce97bafad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:04:55.630578 6963 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
W20240502 08:04:55.630759 12649 consensus_peers.cc:485] T 7fe7134eaf2e4e29ac762afe8855bdac P b79fbd9f226f4270a50e10dce97bafad -> Peer 701bb8dd0643469ab2ead20bab8b0451 (127.6.204.196:37157): Couldn't send request to peer 701bb8dd0643469ab2ead20bab8b0451. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7fe7134eaf2e4e29ac762afe8855bdac. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:04:55.631407 12786 consensus_queue.cc:1035] T 7fe7134eaf2e4e29ac762afe8855bdac P b79fbd9f226f4270a50e10dce97bafad [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240502 08:04:55.632442 6963 kserver.cc:163] Server-wide thread pool size limit: 3276
W20240502 08:04:55.633759 12647 consensus_peers.cc:485] T 7fe7134eaf2e4e29ac762afe8855bdac P b79fbd9f226f4270a50e10dce97bafad -> Peer 7e739706a3cb4269b73b3dc63e28ccfe (127.6.204.193:32777): Couldn't send request to peer 7e739706a3cb4269b73b3dc63e28ccfe. Status: Network error: Client connection negotiation failed: client connection to 127.6.204.193:32777: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:04:55.654297 12575 consensus_peers.cc:485] T 90b6d92c72f44d89a54c7b1b55470f4b P ca3300cf09c04d6cac79551a5ab409ea -> Peer 701bb8dd0643469ab2ead20bab8b0451 (127.6.204.196:37157): Couldn't send request to peer 701bb8dd0643469ab2ead20bab8b0451. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 90b6d92c72f44d89a54c7b1b55470f4b. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:04:55.655906 12683 raft_consensus.cc:1261] T 90b6d92c72f44d89a54c7b1b55470f4b P b79fbd9f226f4270a50e10dce97bafad [term 1 FOLLOWER]: Refusing update from remote peer ca3300cf09c04d6cac79551a5ab409ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:04:55.659879 12813 consensus_queue.cc:1035] T 90b6d92c72f44d89a54c7b1b55470f4b P ca3300cf09c04d6cac79551a5ab409ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240502 08:04:55.670671 6963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:04:55.672309 12785 raft_consensus.cc:2932] T 7fe7134eaf2e4e29ac762afe8855bdac P b79fbd9f226f4270a50e10dce97bafad [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 701bb8dd0643469ab2ead20bab8b0451 (127.6.204.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } } peers { permanent_uuid: "701bb8dd0643469ab2ead20bab8b0451" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 37157 } attrs { promote: true } } }
W20240502 08:04:55.680189 12480 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver 7e739706a3cb4269b73b3dc63e28ccfe not available for placement
W20240502 08:04:55.688653 12573 consensus_peers.cc:485] T 90b6d92c72f44d89a54c7b1b55470f4b P ca3300cf09c04d6cac79551a5ab409ea -> Peer 7e739706a3cb4269b73b3dc63e28ccfe (127.6.204.193:32777): Couldn't send request to peer 7e739706a3cb4269b73b3dc63e28ccfe. Status: Network error: Client connection negotiation failed: client connection to 127.6.204.193:32777: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:04:55.690691 12606 raft_consensus.cc:2932] T 7fe7134eaf2e4e29ac762afe8855bdac P ca3300cf09c04d6cac79551a5ab409ea [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 701bb8dd0643469ab2ead20bab8b0451 (127.6.204.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } } peers { permanent_uuid: "701bb8dd0643469ab2ead20bab8b0451" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 37157 } attrs { promote: true } } }
I20240502 08:04:55.692974 12813 raft_consensus.cc:2932] T 90b6d92c72f44d89a54c7b1b55470f4b P ca3300cf09c04d6cac79551a5ab409ea [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 701bb8dd0643469ab2ead20bab8b0451 (127.6.204.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } } peers { permanent_uuid: "701bb8dd0643469ab2ead20bab8b0451" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 37157 } attrs { promote: true } } }
I20240502 08:04:55.697809 12683 raft_consensus.cc:2932] T 90b6d92c72f44d89a54c7b1b55470f4b P b79fbd9f226f4270a50e10dce97bafad [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 701bb8dd0643469ab2ead20bab8b0451 (127.6.204.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } } peers { permanent_uuid: "701bb8dd0643469ab2ead20bab8b0451" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 37157 } attrs { promote: true } } }
I20240502 08:04:55.703159 12418 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 7fe7134eaf2e4e29ac762afe8855bdac with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 5)
I20240502 08:04:55.707960 12430 catalog_manager.cc:5484] T 7fe7134eaf2e4e29ac762afe8855bdac P ca3300cf09c04d6cac79551a5ab409ea reported cstate change: config changed from index -1 to 2, NON_VOTER 701bb8dd0643469ab2ead20bab8b0451 (127.6.204.196) added. New cstate: current_term: 1 leader_uuid: "b79fbd9f226f4270a50e10dce97bafad" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } } peers { permanent_uuid: "701bb8dd0643469ab2ead20bab8b0451" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 37157 } attrs { promote: true } } }
I20240502 08:04:55.713608 12417 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 90b6d92c72f44d89a54c7b1b55470f4b with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 5)
I20240502 08:04:55.718858 12926 ts_tablet_manager.cc:522] Loading tablet metadata (0/8 complete)
I20240502 08:04:55.724747 12429 catalog_manager.cc:5484] T 90b6d92c72f44d89a54c7b1b55470f4b P ca3300cf09c04d6cac79551a5ab409ea reported cstate change: config changed from index -1 to 2, NON_VOTER 701bb8dd0643469ab2ead20bab8b0451 (127.6.204.196) added. New cstate: current_term: 1 leader_uuid: "ca3300cf09c04d6cac79551a5ab409ea" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "701bb8dd0643469ab2ead20bab8b0451" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 37157 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:04:55.763478 6963 ts_tablet_manager.cc:559] Loaded tablet metadata (8 total tablets, 8 live tablets)
I20240502 08:04:55.763816 6963 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.073s user 0.014s sys 0.015s
I20240502 08:04:55.764133 6963 ts_tablet_manager.cc:574] Registering tablets (0/8 complete)
I20240502 08:04:55.768862 12926 tablet_bootstrap.cc:492] T bde9453a83b8464899af354b5e24806f P 7e739706a3cb4269b73b3dc63e28ccfe: Bootstrap starting.
I20240502 08:04:55.781266 12926 tablet_bootstrap.cc:492] T bde9453a83b8464899af354b5e24806f P 7e739706a3cb4269b73b3dc63e28ccfe: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:04:55.782063 12926 tablet_bootstrap.cc:492] T bde9453a83b8464899af354b5e24806f P 7e739706a3cb4269b73b3dc63e28ccfe: Bootstrap complete.
I20240502 08:04:55.782707 12926 ts_tablet_manager.cc:1367] T bde9453a83b8464899af354b5e24806f P 7e739706a3cb4269b73b3dc63e28ccfe: Time spent bootstrapping tablet: real 0.014s user 0.013s sys 0.000s
I20240502 08:04:55.784626 12926 raft_consensus.cc:348] T bde9453a83b8464899af354b5e24806f P 7e739706a3cb4269b73b3dc63e28ccfe [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:55.785161 12926 raft_consensus.cc:729] T bde9453a83b8464899af354b5e24806f P 7e739706a3cb4269b73b3dc63e28ccfe [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e739706a3cb4269b73b3dc63e28ccfe, State: Initialized, Role: FOLLOWER
I20240502 08:04:55.785722 12926 consensus_queue.cc:260] T bde9453a83b8464899af354b5e24806f P 7e739706a3cb4269b73b3dc63e28ccfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:55.788012 12926 ts_tablet_manager.cc:1398] T bde9453a83b8464899af354b5e24806f P 7e739706a3cb4269b73b3dc63e28ccfe: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
I20240502 08:04:55.788657 12926 tablet_bootstrap.cc:492] T bb44f0c8f9164047ae9d2ce1b67ab9da P 7e739706a3cb4269b73b3dc63e28ccfe: Bootstrap starting.
I20240502 08:04:55.795360 6963 ts_tablet_manager.cc:590] Registered 8 tablets
I20240502 08:04:55.795641 6963 ts_tablet_manager.cc:569] Time spent register tablets: real 0.032s user 0.027s sys 0.004s
I20240502 08:04:55.800282 12926 tablet_bootstrap.cc:492] T bb44f0c8f9164047ae9d2ce1b67ab9da P 7e739706a3cb4269b73b3dc63e28ccfe: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 08:04:55.801049 12926 tablet_bootstrap.cc:492] T bb44f0c8f9164047ae9d2ce1b67ab9da P 7e739706a3cb4269b73b3dc63e28ccfe: Bootstrap complete.
I20240502 08:04:55.801580 12926 ts_tablet_manager.cc:1367] T bb44f0c8f9164047ae9d2ce1b67ab9da P 7e739706a3cb4269b73b3dc63e28ccfe: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.004s
I20240502 08:04:55.803462 12926 raft_consensus.cc:348] T bb44f0c8f9164047ae9d2ce1b67ab9da P 7e739706a3cb4269b73b3dc63e28ccfe [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:55.804453 12926 raft_consensus.cc:729] T bb44f0c8f9164047ae9d2ce1b67ab9da P 7e739706a3cb4269b73b3dc63e28ccfe [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e739706a3cb4269b73b3dc63e28ccfe, State: Initialized, Role: FOLLOWER
I20240502 08:04:55.805303 12926 consensus_queue.cc:260] T bb44f0c8f9164047ae9d2ce1b67ab9da P 7e739706a3cb4269b73b3dc63e28ccfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:55.806891 12926 ts_tablet_manager.cc:1398] T bb44f0c8f9164047ae9d2ce1b67ab9da P 7e739706a3cb4269b73b3dc63e28ccfe: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
I20240502 08:04:55.807561 12926 tablet_bootstrap.cc:492] T 212a1cd8d6244f73b013888b748c3ee8 P 7e739706a3cb4269b73b3dc63e28ccfe: Bootstrap starting.
I20240502 08:04:55.821110 12926 tablet_bootstrap.cc:492] T 212a1cd8d6244f73b013888b748c3ee8 P 7e739706a3cb4269b73b3dc63e28ccfe: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 08:04:55.822296 12926 tablet_bootstrap.cc:492] T 212a1cd8d6244f73b013888b748c3ee8 P 7e739706a3cb4269b73b3dc63e28ccfe: Bootstrap complete.
I20240502 08:04:55.823130 12926 ts_tablet_manager.cc:1367] T 212a1cd8d6244f73b013888b748c3ee8 P 7e739706a3cb4269b73b3dc63e28ccfe: Time spent bootstrapping tablet: real 0.016s user 0.011s sys 0.004s
I20240502 08:04:55.826756 12926 raft_consensus.cc:348] T 212a1cd8d6244f73b013888b748c3ee8 P 7e739706a3cb4269b73b3dc63e28ccfe [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:55.828024 12926 raft_consensus.cc:729] T 212a1cd8d6244f73b013888b748c3ee8 P 7e739706a3cb4269b73b3dc63e28ccfe [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e739706a3cb4269b73b3dc63e28ccfe, State: Initialized, Role: FOLLOWER
I20240502 08:04:55.828927 12926 consensus_queue.cc:260] T 212a1cd8d6244f73b013888b748c3ee8 P 7e739706a3cb4269b73b3dc63e28ccfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:55.831575 12926 ts_tablet_manager.cc:1398] T 212a1cd8d6244f73b013888b748c3ee8 P 7e739706a3cb4269b73b3dc63e28ccfe: Time spent starting tablet: real 0.008s user 0.004s sys 0.003s
I20240502 08:04:55.832581 12926 tablet_bootstrap.cc:492] T 1abb47c8c0a642b097ae6327e60ec0e0 P 7e739706a3cb4269b73b3dc63e28ccfe: Bootstrap starting.
I20240502 08:04:55.850505 12926 tablet_bootstrap.cc:492] T 1abb47c8c0a642b097ae6327e60ec0e0 P 7e739706a3cb4269b73b3dc63e28ccfe: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:04:55.850893 6963 rpc_server.cc:307] RPC server started. Bound to: 127.6.204.193:32777
I20240502 08:04:55.851104 12988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.204.193:32777 every 8 connection(s)
I20240502 08:04:55.851492 12926 tablet_bootstrap.cc:492] T 1abb47c8c0a642b097ae6327e60ec0e0 P 7e739706a3cb4269b73b3dc63e28ccfe: Bootstrap complete.
I20240502 08:04:55.852053 12926 ts_tablet_manager.cc:1367] T 1abb47c8c0a642b097ae6327e60ec0e0 P 7e739706a3cb4269b73b3dc63e28ccfe: Time spent bootstrapping tablet: real 0.020s user 0.017s sys 0.000s
I20240502 08:04:55.854014 12926 raft_consensus.cc:348] T 1abb47c8c0a642b097ae6327e60ec0e0 P 7e739706a3cb4269b73b3dc63e28ccfe [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:55.854734 12926 raft_consensus.cc:729] T 1abb47c8c0a642b097ae6327e60ec0e0 P 7e739706a3cb4269b73b3dc63e28ccfe [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e739706a3cb4269b73b3dc63e28ccfe, State: Initialized, Role: FOLLOWER
I20240502 08:04:55.855435 12926 consensus_queue.cc:260] T 1abb47c8c0a642b097ae6327e60ec0e0 P 7e739706a3cb4269b73b3dc63e28ccfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:55.857399 12926 ts_tablet_manager.cc:1398] T 1abb47c8c0a642b097ae6327e60ec0e0 P 7e739706a3cb4269b73b3dc63e28ccfe: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240502 08:04:55.858189 12926 tablet_bootstrap.cc:492] T 7fe7134eaf2e4e29ac762afe8855bdac P 7e739706a3cb4269b73b3dc63e28ccfe: Bootstrap starting.
I20240502 08:04:55.869504 12989 heartbeater.cc:347] Connected to a master server at 127.6.204.254:32811
I20240502 08:04:55.869879 12989 heartbeater.cc:464] Registering TS with master...
I20240502 08:04:55.870590 12989 heartbeater.cc:511] Master 127.6.204.254:32811 requested a full tablet report, sending...
I20240502 08:04:55.871879 12926 tablet_bootstrap.cc:492] T 7fe7134eaf2e4e29ac762afe8855bdac P 7e739706a3cb4269b73b3dc63e28ccfe: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:04:55.872570 12926 tablet_bootstrap.cc:492] T 7fe7134eaf2e4e29ac762afe8855bdac P 7e739706a3cb4269b73b3dc63e28ccfe: Bootstrap complete.
I20240502 08:04:55.873073 12926 ts_tablet_manager.cc:1367] T 7fe7134eaf2e4e29ac762afe8855bdac P 7e739706a3cb4269b73b3dc63e28ccfe: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20240502 08:04:55.874997 12926 raft_consensus.cc:348] T 7fe7134eaf2e4e29ac762afe8855bdac P 7e739706a3cb4269b73b3dc63e28ccfe [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:55.875192 12429 ts_manager.cc:194] Re-registered known tserver with Master: 7e739706a3cb4269b73b3dc63e28ccfe (127.6.204.193:32777)
I20240502 08:04:55.875520 12926 raft_consensus.cc:729] T 7fe7134eaf2e4e29ac762afe8855bdac P 7e739706a3cb4269b73b3dc63e28ccfe [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e739706a3cb4269b73b3dc63e28ccfe, State: Initialized, Role: FOLLOWER
I20240502 08:04:55.876149 12926 consensus_queue.cc:260] T 7fe7134eaf2e4e29ac762afe8855bdac P 7e739706a3cb4269b73b3dc63e28ccfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:55.877583 12926 ts_tablet_manager.cc:1398] T 7fe7134eaf2e4e29ac762afe8855bdac P 7e739706a3cb4269b73b3dc63e28ccfe: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240502 08:04:55.878221 12926 tablet_bootstrap.cc:492] T 0f00e1259b934b7a904dbc7dc4a434ae P 7e739706a3cb4269b73b3dc63e28ccfe: Bootstrap starting.
I20240502 08:04:55.883652 12429 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47806
I20240502 08:04:55.886657 12989 heartbeater.cc:503] Master 127.6.204.254:32811 was elected leader, sending a full tablet report...
I20240502 08:04:55.888702 12926 tablet_bootstrap.cc:492] T 0f00e1259b934b7a904dbc7dc4a434ae P 7e739706a3cb4269b73b3dc63e28ccfe: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 08:04:55.889624 12926 tablet_bootstrap.cc:492] T 0f00e1259b934b7a904dbc7dc4a434ae P 7e739706a3cb4269b73b3dc63e28ccfe: Bootstrap complete.
I20240502 08:04:55.890234 12926 ts_tablet_manager.cc:1367] T 0f00e1259b934b7a904dbc7dc4a434ae P 7e739706a3cb4269b73b3dc63e28ccfe: Time spent bootstrapping tablet: real 0.012s user 0.012s sys 0.000s
I20240502 08:04:55.891942 12926 raft_consensus.cc:348] T 0f00e1259b934b7a904dbc7dc4a434ae P 7e739706a3cb4269b73b3dc63e28ccfe [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:55.892524 12926 raft_consensus.cc:729] T 0f00e1259b934b7a904dbc7dc4a434ae P 7e739706a3cb4269b73b3dc63e28ccfe [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e739706a3cb4269b73b3dc63e28ccfe, State: Initialized, Role: FOLLOWER
I20240502 08:04:55.893038 12926 consensus_queue.cc:260] T 0f00e1259b934b7a904dbc7dc4a434ae P 7e739706a3cb4269b73b3dc63e28ccfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:55.894522 12926 ts_tablet_manager.cc:1398] T 0f00e1259b934b7a904dbc7dc4a434ae P 7e739706a3cb4269b73b3dc63e28ccfe: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240502 08:04:55.895095 12926 tablet_bootstrap.cc:492] T 9badc171df9a47f9bb33b775dea31604 P 7e739706a3cb4269b73b3dc63e28ccfe: Bootstrap starting.
I20240502 08:04:55.904908 12926 tablet_bootstrap.cc:492] T 9badc171df9a47f9bb33b775dea31604 P 7e739706a3cb4269b73b3dc63e28ccfe: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:04:55.905599 12926 tablet_bootstrap.cc:492] T 9badc171df9a47f9bb33b775dea31604 P 7e739706a3cb4269b73b3dc63e28ccfe: Bootstrap complete.
I20240502 08:04:55.906204 12926 ts_tablet_manager.cc:1367] T 9badc171df9a47f9bb33b775dea31604 P 7e739706a3cb4269b73b3dc63e28ccfe: Time spent bootstrapping tablet: real 0.011s user 0.011s sys 0.000s
I20240502 08:04:55.907723 12926 raft_consensus.cc:348] T 9badc171df9a47f9bb33b775dea31604 P 7e739706a3cb4269b73b3dc63e28ccfe [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:55.908380 12926 raft_consensus.cc:729] T 9badc171df9a47f9bb33b775dea31604 P 7e739706a3cb4269b73b3dc63e28ccfe [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e739706a3cb4269b73b3dc63e28ccfe, State: Initialized, Role: FOLLOWER
I20240502 08:04:55.909019 12926 consensus_queue.cc:260] T 9badc171df9a47f9bb33b775dea31604 P 7e739706a3cb4269b73b3dc63e28ccfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:55.911053 12926 ts_tablet_manager.cc:1398] T 9badc171df9a47f9bb33b775dea31604 P 7e739706a3cb4269b73b3dc63e28ccfe: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240502 08:04:55.911795 12926 tablet_bootstrap.cc:492] T 90b6d92c72f44d89a54c7b1b55470f4b P 7e739706a3cb4269b73b3dc63e28ccfe: Bootstrap starting.
I20240502 08:04:55.920348 12953 tablet_service.cc:1514] Processing DeleteTablet for tablet bb44f0c8f9164047ae9d2ce1b67ab9da with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:04:55 UTC) from {username='slave'} at 127.0.0.1:39498
I20240502 08:04:55.921008 12952 tablet_service.cc:1514] Processing DeleteTablet for tablet 9badc171df9a47f9bb33b775dea31604 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:04:55 UTC) from {username='slave'} at 127.0.0.1:39498
I20240502 08:04:55.923553 12997 tablet_replica.cc:330] stopping tablet replica
I20240502 08:04:55.924212 12997 raft_consensus.cc:2229] T bb44f0c8f9164047ae9d2ce1b67ab9da P 7e739706a3cb4269b73b3dc63e28ccfe [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:04:55.924566 12997 pending_rounds.cc:62] T bb44f0c8f9164047ae9d2ce1b67ab9da P 7e739706a3cb4269b73b3dc63e28ccfe: Trying to abort 1 pending ops.
I20240502 08:04:55.924779 12997 pending_rounds.cc:68] T bb44f0c8f9164047ae9d2ce1b67ab9da P 7e739706a3cb4269b73b3dc63e28ccfe: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023153536508051456 op_type: NO_OP noop_request { }
I20240502 08:04:55.925153 12997 raft_consensus.cc:2866] T bb44f0c8f9164047ae9d2ce1b67ab9da P 7e739706a3cb4269b73b3dc63e28ccfe [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20240502 08:04:55.925493 12997 raft_consensus.cc:2258] T bb44f0c8f9164047ae9d2ce1b67ab9da P 7e739706a3cb4269b73b3dc63e28ccfe [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:04:55.926297 12926 tablet_bootstrap.cc:492] T 90b6d92c72f44d89a54c7b1b55470f4b P 7e739706a3cb4269b73b3dc63e28ccfe: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:04:55.927064 12926 tablet_bootstrap.cc:492] T 90b6d92c72f44d89a54c7b1b55470f4b P 7e739706a3cb4269b73b3dc63e28ccfe: Bootstrap complete.
I20240502 08:04:55.927609 12926 ts_tablet_manager.cc:1367] T 90b6d92c72f44d89a54c7b1b55470f4b P 7e739706a3cb4269b73b3dc63e28ccfe: Time spent bootstrapping tablet: real 0.016s user 0.011s sys 0.004s
I20240502 08:04:55.927862 12997 ts_tablet_manager.cc:1875] T bb44f0c8f9164047ae9d2ce1b67ab9da P 7e739706a3cb4269b73b3dc63e28ccfe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:04:55.929416 12926 raft_consensus.cc:348] T 90b6d92c72f44d89a54c7b1b55470f4b P 7e739706a3cb4269b73b3dc63e28ccfe [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:55.929999 12926 raft_consensus.cc:729] T 90b6d92c72f44d89a54c7b1b55470f4b P 7e739706a3cb4269b73b3dc63e28ccfe [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e739706a3cb4269b73b3dc63e28ccfe, State: Initialized, Role: FOLLOWER
I20240502 08:04:55.930545 12926 consensus_queue.cc:260] T 90b6d92c72f44d89a54c7b1b55470f4b P 7e739706a3cb4269b73b3dc63e28ccfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:55.932228 12926 ts_tablet_manager.cc:1398] T 90b6d92c72f44d89a54c7b1b55470f4b P 7e739706a3cb4269b73b3dc63e28ccfe: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240502 08:04:55.938777 12997 ts_tablet_manager.cc:1888] T bb44f0c8f9164047ae9d2ce1b67ab9da P 7e739706a3cb4269b73b3dc63e28ccfe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:04:55.939203 12997 log.cc:1196] T bb44f0c8f9164047ae9d2ce1b67ab9da P 7e739706a3cb4269b73b3dc63e28ccfe: Deleting WAL directory at /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714636933625983-6963-0/minicluster-data/ts-0-root/wals/bb44f0c8f9164047ae9d2ce1b67ab9da
I20240502 08:04:55.939999 12997 ts_tablet_manager.cc:1909] T bb44f0c8f9164047ae9d2ce1b67ab9da P 7e739706a3cb4269b73b3dc63e28ccfe: Deleting consensus metadata
I20240502 08:04:55.942317 12997 tablet_replica.cc:330] stopping tablet replica
I20240502 08:04:55.942324 12417 catalog_manager.cc:4830] TS 7e739706a3cb4269b73b3dc63e28ccfe (127.6.204.193:32777): tablet bb44f0c8f9164047ae9d2ce1b67ab9da (table dugtrio [id=d58d27d9f8724a5ca12188de828572b7]) successfully deleted
I20240502 08:04:55.943018 12997 raft_consensus.cc:2229] T 9badc171df9a47f9bb33b775dea31604 P 7e739706a3cb4269b73b3dc63e28ccfe [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:04:55.943434 12997 raft_consensus.cc:2258] T 9badc171df9a47f9bb33b775dea31604 P 7e739706a3cb4269b73b3dc63e28ccfe [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:04:55.945271 12997 ts_tablet_manager.cc:1875] T 9badc171df9a47f9bb33b775dea31604 P 7e739706a3cb4269b73b3dc63e28ccfe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:04:55.954741 12997 ts_tablet_manager.cc:1888] T 9badc171df9a47f9bb33b775dea31604 P 7e739706a3cb4269b73b3dc63e28ccfe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:04:55.955032 12997 log.cc:1196] T 9badc171df9a47f9bb33b775dea31604 P 7e739706a3cb4269b73b3dc63e28ccfe: Deleting WAL directory at /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714636933625983-6963-0/minicluster-data/ts-0-root/wals/9badc171df9a47f9bb33b775dea31604
I20240502 08:04:55.955621 12997 ts_tablet_manager.cc:1909] T 9badc171df9a47f9bb33b775dea31604 P 7e739706a3cb4269b73b3dc63e28ccfe: Deleting consensus metadata
I20240502 08:04:55.957736 12417 catalog_manager.cc:4830] TS 7e739706a3cb4269b73b3dc63e28ccfe (127.6.204.193:32777): tablet 9badc171df9a47f9bb33b775dea31604 (table dugtrio [id=d58d27d9f8724a5ca12188de828572b7]) successfully deleted
I20240502 08:04:55.959388 12606 raft_consensus.cc:1261] T 1abb47c8c0a642b097ae6327e60ec0e0 P ca3300cf09c04d6cac79551a5ab409ea [term 2 FOLLOWER]: Refusing update from remote peer b79fbd9f226f4270a50e10dce97bafad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:04:55.960434 12953 tablet_service.cc:1514] Processing DeleteTablet for tablet 212a1cd8d6244f73b013888b748c3ee8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:04:55 UTC) from {username='slave'} at 127.0.0.1:39498
I20240502 08:04:55.960605 12786 consensus_queue.cc:1035] T 1abb47c8c0a642b097ae6327e60ec0e0 P b79fbd9f226f4270a50e10dce97bafad [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240502 08:04:55.961081 12997 tablet_replica.cc:330] stopping tablet replica
I20240502 08:04:55.961843 12997 raft_consensus.cc:2229] T 212a1cd8d6244f73b013888b748c3ee8 P 7e739706a3cb4269b73b3dc63e28ccfe [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:04:55.962227 12997 pending_rounds.cc:62] T 212a1cd8d6244f73b013888b748c3ee8 P 7e739706a3cb4269b73b3dc63e28ccfe: Trying to abort 1 pending ops.
I20240502 08:04:55.962412 12997 pending_rounds.cc:68] T 212a1cd8d6244f73b013888b748c3ee8 P 7e739706a3cb4269b73b3dc63e28ccfe: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023153537029038080 op_type: NO_OP noop_request { }
I20240502 08:04:55.962734 12997 raft_consensus.cc:2866] T 212a1cd8d6244f73b013888b748c3ee8 P 7e739706a3cb4269b73b3dc63e28ccfe [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20240502 08:04:55.963032 12997 raft_consensus.cc:2258] T 212a1cd8d6244f73b013888b748c3ee8 P 7e739706a3cb4269b73b3dc63e28ccfe [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:04:55.965464 12997 ts_tablet_manager.cc:1875] T 212a1cd8d6244f73b013888b748c3ee8 P 7e739706a3cb4269b73b3dc63e28ccfe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:04:55.981118 12997 ts_tablet_manager.cc:1888] T 212a1cd8d6244f73b013888b748c3ee8 P 7e739706a3cb4269b73b3dc63e28ccfe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:04:55.981531 12997 log.cc:1196] T 212a1cd8d6244f73b013888b748c3ee8 P 7e739706a3cb4269b73b3dc63e28ccfe: Deleting WAL directory at /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714636933625983-6963-0/minicluster-data/ts-0-root/wals/212a1cd8d6244f73b013888b748c3ee8
I20240502 08:04:55.982267 12997 ts_tablet_manager.cc:1909] T 212a1cd8d6244f73b013888b748c3ee8 P 7e739706a3cb4269b73b3dc63e28ccfe: Deleting consensus metadata
I20240502 08:04:55.982316 12963 raft_consensus.cc:3037] T 1abb47c8c0a642b097ae6327e60ec0e0 P 7e739706a3cb4269b73b3dc63e28ccfe [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:04:55.984580 12417 catalog_manager.cc:4830] TS 7e739706a3cb4269b73b3dc63e28ccfe (127.6.204.193:32777): tablet 212a1cd8d6244f73b013888b748c3ee8 (table dugtrio [id=d58d27d9f8724a5ca12188de828572b7]) successfully deleted
I20240502 08:04:55.987247 12963 raft_consensus.cc:1261] T 1abb47c8c0a642b097ae6327e60ec0e0 P 7e739706a3cb4269b73b3dc63e28ccfe [term 2 FOLLOWER]: Refusing update from remote peer b79fbd9f226f4270a50e10dce97bafad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:04:55.988461 12953 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f00e1259b934b7a904dbc7dc4a434ae with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:04:55 UTC) from {username='slave'} at 127.0.0.1:39498
I20240502 08:04:55.988456 12785 consensus_queue.cc:1035] T 1abb47c8c0a642b097ae6327e60ec0e0 P b79fbd9f226f4270a50e10dce97bafad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240502 08:04:55.989229 12997 tablet_replica.cc:330] stopping tablet replica
I20240502 08:04:55.989852 12997 raft_consensus.cc:2229] T 0f00e1259b934b7a904dbc7dc4a434ae P 7e739706a3cb4269b73b3dc63e28ccfe [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:04:55.990337 12997 pending_rounds.cc:62] T 0f00e1259b934b7a904dbc7dc4a434ae P 7e739706a3cb4269b73b3dc63e28ccfe: Trying to abort 1 pending ops.
I20240502 08:04:55.990641 12997 pending_rounds.cc:68] T 0f00e1259b934b7a904dbc7dc4a434ae P 7e739706a3cb4269b73b3dc63e28ccfe: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023153536868663296 op_type: NO_OP noop_request { }
I20240502 08:04:55.991189 12997 raft_consensus.cc:2866] T 0f00e1259b934b7a904dbc7dc4a434ae P 7e739706a3cb4269b73b3dc63e28ccfe [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20240502 08:04:55.991492 12997 raft_consensus.cc:2258] T 0f00e1259b934b7a904dbc7dc4a434ae P 7e739706a3cb4269b73b3dc63e28ccfe [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:04:55.994019 12997 ts_tablet_manager.cc:1875] T 0f00e1259b934b7a904dbc7dc4a434ae P 7e739706a3cb4269b73b3dc63e28ccfe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:04:56.007588 12997 ts_tablet_manager.cc:1888] T 0f00e1259b934b7a904dbc7dc4a434ae P 7e739706a3cb4269b73b3dc63e28ccfe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:04:56.007934 12997 log.cc:1196] T 0f00e1259b934b7a904dbc7dc4a434ae P 7e739706a3cb4269b73b3dc63e28ccfe: Deleting WAL directory at /tmp/dist-test-taskY1qqxp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714636933625983-6963-0/minicluster-data/ts-0-root/wals/0f00e1259b934b7a904dbc7dc4a434ae
I20240502 08:04:56.008692 12997 ts_tablet_manager.cc:1909] T 0f00e1259b934b7a904dbc7dc4a434ae P 7e739706a3cb4269b73b3dc63e28ccfe: Deleting consensus metadata
I20240502 08:04:56.011034 12417 catalog_manager.cc:4830] TS 7e739706a3cb4269b73b3dc63e28ccfe (127.6.204.193:32777): tablet 0f00e1259b934b7a904dbc7dc4a434ae (table dugtrio [id=d58d27d9f8724a5ca12188de828572b7]) successfully deleted
I20240502 08:04:56.032079 13003 ts_tablet_manager.cc:902] T bde9453a83b8464899af354b5e24806f P 701bb8dd0643469ab2ead20bab8b0451: Initiating tablet copy from peer b79fbd9f226f4270a50e10dce97bafad (127.6.204.195:46407)
F20240502 08:04:56.032891 13003 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (init.queue_time_us, init.queue_time_us) (init.run_cpu_time_us, init.run_cpu_time_us) (init.run_wall_time_us, init.run_wall_time_us) (leader-initialization.queue_time_us, leader-initialization.queue_time_us) (leader-initialization.run_cpu_time_us, leader-initialization.run_cpu_time_us) (leader-initialization.run_wall_time_us, leader-initialization.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-20f890980e404e5ea6ec8995c97ed78c.queue_time_us, tablet-download-pool-20f890980e404e5ea6ec8995c97ed78c.queue_time_us) (tablet-download-pool-20f890980e404e5ea6ec8995c97ed78c.run_cpu_time_us, tablet-download-pool-20f890980e404e5ea6ec8995c97ed78c.run_cpu_time_us) (tablet-download-pool-20f890980e404e5ea6ec8995c97ed78c.run_wall_time_us, tablet-download-pool-20f890980e404e5ea6ec8995c97ed78c.run_wall_time_us) (tablet-download-pool-33dd558b4b60478c97aad83f82b9f1e5.queue_time_us, tablet-download-pool-33dd558b4b60478c97aad83f82b9f1e5.queue_time_us) (tablet-download-pool-33dd558b4b60478c97aad83f82b9f1e5.run_cpu_time_us, tablet-download-pool-33dd558b4b60478c97aad83f82b9f1e5.run_cpu_time_us) (tablet-download-pool-33dd558b4b60478c97aad83f82b9f1e5.run_wall_time_us, tablet-download-pool-33dd558b4b60478c97aad83f82b9f1e5.run_wall_time_us) (tablet-download-pool-385fa1c24869499ab21a9a6852f82494.queue_time_us, tablet-download-pool-385fa1c24869499ab21a9a6852f82494.queue_time_us) (tablet-download-pool-385fa1c24869499ab21a9a6852f82494.run_cpu_time_us, tablet-download-pool-385fa1c24869499ab21a9a6852f82494.run_cpu_time_us) (tablet-download-pool-385fa1c24869499ab21a9a6852f82494.run_wall_time_us, tablet-download-pool-385fa1c24869499ab21a9a6852f82494.run_wall_time_us) (tablet-download-pool-743666c9f9a144978e534923cc63222e.queue_time_us, tablet-download-pool-743666c9f9a144978e534923cc63222e.queue_time_us) (tablet-download-pool-743666c9f9a144978e534923cc63222e.run_cpu_time_us, tablet-download-pool-743666c9f9a144978e534923cc63222e.run_cpu_time_us) (tablet-download-pool-743666c9f9a144978e534923cc63222e.run_wall_time_us, tablet-download-pool-743666c9f9a144978e534923cc63222e.run_wall_time_us) (tablet-download-pool-8ad9d4c321a449539ca014fc4e89005d.queue_time_us, tablet-download-pool-8ad9d4c321a449539ca014fc4e89005d.queue_time_us) (tablet-download-pool-8ad9d4c321a449539ca014fc4e89005d.run_cpu_time_us, tablet-download-pool-8ad9d4c321a449539ca014fc4e89005d.run_cpu_time_us) (tablet-download-pool-8ad9d4c321a449539ca014fc4e89005d.run_wall_time_us, tablet-download-pool-8ad9d4c321a449539ca014fc4e89005d.run_wall_time_us) (tablet-download-pool-8c2569709c52401cae6702e5e931ca83.queue_time_us, tablet-download-pool-8c2569709c52401cae6702e5e931ca83.queue_time_us) (tablet-download-pool-8c2569709c52401cae6702e5e931ca83.run_cpu_time_us, tablet-download-pool-8c2569709c52401cae6702e5e931ca83.run_cpu_time_us) (tablet-download-pool-8c2569709c52401cae6702e5e931ca83.run_wall_time_us, tablet-download-pool-8c2569709c52401cae6702e5e931ca83.run_wall_time_us) (tablet-download-pool-af8197b1de49493a966b8ba1146ada68.queue_time_us, tablet-download-pool-af8197b1de49493a966b8ba1146ada68.queue_time_us) (tablet-download-pool-af8197b1de49493a966b8ba1146ada68.run_cpu_time_us, tablet-download-pool-af8197b1de49493a966b8ba1146ada68.run_cpu_time_us) (tablet-download-pool-af8197b1de49493a966b8ba1146ada68.run_wall_time_us, tablet-download-pool-af8197b1de49493a966b8ba1146ada68.run_wall_time_us) (tablet-download-pool-b69b32e4d5b04f00956ef81511e7c9eb.queue_time_us, tablet-download-pool-b69b32e4d5b04f00956ef81511e7c9eb.queue_time_us) (tablet-download-pool-b69b32e4d5b04f00956ef81511e7c9eb.run_cpu_time_us, tablet-download-pool-b69b32e4d5b04f00956ef81511e7c9eb.run_cpu_time_us) (tablet-download-pool-b69b32e4d5b04f00956ef81511e7c9eb.run_wall_time_us, tablet-download-pool-b69b32e4d5b04f00956ef81511e7c9eb.run_wall_time_us) (tablet-download-pool-b7961f100fcd49f581e2b1bfa1f1ec05.queue_time_us, tablet-download-pool-b7961f100fcd49f581e2b1bfa1f1ec05.queue_time_us) (tablet-download-pool-b7961f100fcd49f581e2b1bfa1f1ec05.run_cpu_time_us, tablet-download-pool-b7961f100fcd49f581e2b1bfa1f1ec05.run_cpu_time_us) (tablet-download-pool-b7961f100fcd49f581e2b1bfa1f1ec05.run_wall_time_us, tablet-download-pool-b7961f100fcd49f581e2b1bfa1f1ec05.run_wall_time_us) (tablet-download-pool-bde9453a83b8464899af354b5e24806f.queue_time_us, tablet-download-pool-bde9453a83b8464899af354b5e24806f.queue_time_us) (tablet-download-pool-d117f9452626438cb18dda39df6eb6ed.queue_time_us, tablet-download-pool-d117f9452626438cb18dda39df6eb6ed.queue_time_us) (tablet-download-pool-d117f9452626438cb18dda39df6eb6ed.run_cpu_time_us, tablet-download-pool-d117f9452626438cb18dda39df6eb6ed.run_cpu_time_us) (tablet-download-pool-d117f9452626438cb18dda39df6eb6ed.run_wall_time_us, tablet-download-pool-d117f9452626438cb18dda39df6eb6ed.run_wall_time_us) (tablet-download-pool-f3666c62e67047bb9393d96859b539eb.queue_time_us, tablet-download-pool-f3666c62e67047bb9393d96859b539eb.queue_time_us) (tablet-download-pool-f3666c62e67047bb9393d96859b539eb.run_cpu_time_us, tablet-download-pool-f3666c62e67047bb9393d96859b539eb.run_cpu_time_us) (tablet-download-pool-f3666c62e67047bb9393d96859b539eb.run_wall_time_us, tablet-download-pool-f3666c62e67047bb9393d96859b539eb.run_wall_time_us) (tablet-download-pool-f8a4589c0a9b417188a2d884011b0240.queue_time_us, tablet-download-pool-f8a4589c0a9b417188a2d884011b0240.queue_time_us) (tablet-download-pool-f8a4589c0a9b417188a2d884011b0240.run_cpu_time_us, tablet-download-pool-f8a4589c0a9b417188a2d884011b0240.run_cpu_time_us) (tablet-download-pool-f8a4589c0a9b417188a2d884011b0240.run_wall_time_us, tablet-download-pool-f8a4589c0a9b417188a2d884011b0240.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
*** Aborted at 1714637096 (unix time) try "date -d @1714637096" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e800001b33) received by PID 6963 (TID 0x7f5a5a9fb700) from PID 6963; stack trace: ***
@ 0x2e41a0 __tsan::CallUserSignalHandler() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1915
@ 0x2e6683 rtl_sigaction() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:2005
@ 0x7f5affed4980 (unknown) at ??:0
I20240502 08:04:56.103768 12963 raft_consensus.cc:2932] T 90b6d92c72f44d89a54c7b1b55470f4b P 7e739706a3cb4269b73b3dc63e28ccfe [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 701bb8dd0643469ab2ead20bab8b0451 (127.6.204.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } } peers { permanent_uuid: "701bb8dd0643469ab2ead20bab8b0451" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 37157 } attrs { promote: true } } }
@ 0x7f5af9448fb7 gsignal at ??:0
I20240502 08:04:56.120729 13007 ts_tablet_manager.cc:902] T 7fe7134eaf2e4e29ac762afe8855bdac P 701bb8dd0643469ab2ead20bab8b0451: Initiating tablet copy from peer b79fbd9f226f4270a50e10dce97bafad (127.6.204.195:46407)
@ 0x7f5af944a921 abort at ??:0
@ 0x2ea196 abort at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:263
I20240502 08:04:56.152515 12963 raft_consensus.cc:2932] T bde9453a83b8464899af354b5e24806f P 7e739706a3cb4269b73b3dc63e28ccfe [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 701bb8dd0643469ab2ead20bab8b0451 (127.6.204.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } } peers { permanent_uuid: "701bb8dd0643469ab2ead20bab8b0451" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 37157 } attrs { promote: true } } }
@ 0x7f5afc23eda4 google::LogMessage::SendToLog() at ??:0
@ 0x7f5afc23f950 google::LogMessage::Flush() at ??:0
I20240502 08:04:56.205471 12963 raft_consensus.cc:2932] T 7fe7134eaf2e4e29ac762afe8855bdac P 7e739706a3cb4269b73b3dc63e28ccfe [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 701bb8dd0643469ab2ead20bab8b0451 (127.6.204.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } } peers { permanent_uuid: "701bb8dd0643469ab2ead20bab8b0451" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 37157 } attrs { promote: true } } }
@ 0x7f5afc244a8b google::LogMessageFatal::~LogMessageFatal() at ??:0
I20240502 08:04:56.218250 13014 ts_tablet_manager.cc:902] T 90b6d92c72f44d89a54c7b1b55470f4b P 701bb8dd0643469ab2ead20bab8b0451: Initiating tablet copy from peer ca3300cf09c04d6cac79551a5ab409ea (127.6.204.194:39897)
@ 0x7f5afcbcc3c7 kudu::TraceMetrics::InternName() at ??:0
@ 0x7f5afcba294f kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f5afcba00a5 kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f5b049c0422 kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f5b049c6c89 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7f5b04a5c4ef kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7f5b04a8d3cf kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f5b04a731ce kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
@ 0x7f5b04a7317a _ZNSt3__18__invokeIRZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBENS_8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEE4$_12JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOSH_DpOSI_ at ??:0
@ 0x7f5b04a7310a std::__1::__invoke_void_return_wrapper<>::__call<>() at ??:0
@ 0x7f5b04a730d2 std::__1::__function::__alloc_func<>::operator()() at ??:0
@ 0x7f5b04a722fe std::__1::__function::__func<>::operator()() at ??:0
@ 0x7f5b05e18da5 std::__1::__function::__value_func<>::operator()() at ??:0
@ 0x7f5b05e18bda std::__1::function<>::operator()() at ??:0
I20240502 08:04:56.535614 12897 heartbeater.cc:503] Master 127.6.204.254:32811 was elected leader, sending a full tablet report...
@ 0x7f5afcba4237 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f5afcba7692 kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f5afcba764a _ZNSt3__18__invokeIRZN4kudu10ThreadPool12CreateThreadEvE3$_2JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS5_DpOS6_ at ??:0
@ 0x7f5afcba75da std::__1::__invoke_void_return_wrapper<>::__call<>() at ??:0
I20240502 08:04:56.598404 12606 consensus_queue.cc:237] T 90b6d92c72f44d89a54c7b1b55470f4b P ca3300cf09c04d6cac79551a5ab409ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:56.603493 12683 raft_consensus.cc:1261] T 90b6d92c72f44d89a54c7b1b55470f4b P b79fbd9f226f4270a50e10dce97bafad [term 1 FOLLOWER]: Refusing update from remote peer ca3300cf09c04d6cac79551a5ab409ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:04:56.604106 12963 raft_consensus.cc:1261] T 90b6d92c72f44d89a54c7b1b55470f4b P 7e739706a3cb4269b73b3dc63e28ccfe [term 1 FOLLOWER]: Refusing update from remote peer ca3300cf09c04d6cac79551a5ab409ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:04:56.604674 13036 consensus_queue.cc:1035] T 90b6d92c72f44d89a54c7b1b55470f4b P ca3300cf09c04d6cac79551a5ab409ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:04:56.605763 13037 consensus_queue.cc:1035] T 90b6d92c72f44d89a54c7b1b55470f4b P ca3300cf09c04d6cac79551a5ab409ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:04:56.611306 13036 raft_consensus.cc:2932] T 90b6d92c72f44d89a54c7b1b55470f4b P ca3300cf09c04d6cac79551a5ab409ea [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 701bb8dd0643469ab2ead20bab8b0451 (127.6.204.196) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } } }
I20240502 08:04:56.612998 12963 raft_consensus.cc:2932] T 90b6d92c72f44d89a54c7b1b55470f4b P 7e739706a3cb4269b73b3dc63e28ccfe [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 701bb8dd0643469ab2ead20bab8b0451 (127.6.204.196) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } } }
@ 0x7f5afcba75a2 std::__1::__function::__alloc_func<>::operator()() at ??:0
I20240502 08:04:56.623121 12682 raft_consensus.cc:2932] T 90b6d92c72f44d89a54c7b1b55470f4b P b79fbd9f226f4270a50e10dce97bafad [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 701bb8dd0643469ab2ead20bab8b0451 (127.6.204.196) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } } }
I20240502 08:04:56.627082 12417 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 90b6d92c72f44d89a54c7b1b55470f4b with cas_config_opid_index 2: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:04:56.635324 12432 catalog_manager.cc:5484] T 90b6d92c72f44d89a54c7b1b55470f4b P ca3300cf09c04d6cac79551a5ab409ea reported cstate change: config changed from index 2 to 3, NON_VOTER 701bb8dd0643469ab2ead20bab8b0451 (127.6.204.196) evicted. New cstate: current_term: 1 leader_uuid: "ca3300cf09c04d6cac79551a5ab409ea" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } health_report { overall_health: HEALTHY } } }
I20240502 08:04:56.641584 12682 consensus_queue.cc:237] T 7fe7134eaf2e4e29ac762afe8855bdac P b79fbd9f226f4270a50e10dce97bafad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:56.648494 12606 raft_consensus.cc:1261] T 7fe7134eaf2e4e29ac762afe8855bdac P ca3300cf09c04d6cac79551a5ab409ea [term 1 FOLLOWER]: Refusing update from remote peer b79fbd9f226f4270a50e10dce97bafad: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
@ 0x7f5afcba689e std::__1::__function::__func<>::operator()() at ??:0
I20240502 08:04:56.648882 12963 raft_consensus.cc:1261] T 7fe7134eaf2e4e29ac762afe8855bdac P 7e739706a3cb4269b73b3dc63e28ccfe [term 1 FOLLOWER]: Refusing update from remote peer b79fbd9f226f4270a50e10dce97bafad: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:04:56.650908 12786 consensus_queue.cc:1035] T 7fe7134eaf2e4e29ac762afe8855bdac P b79fbd9f226f4270a50e10dce97bafad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 08:04:56.652913 12786 consensus_queue.cc:1035] T 7fe7134eaf2e4e29ac762afe8855bdac P b79fbd9f226f4270a50e10dce97bafad [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:04:56.666013 12785 raft_consensus.cc:2932] T 7fe7134eaf2e4e29ac762afe8855bdac P b79fbd9f226f4270a50e10dce97bafad [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 701bb8dd0643469ab2ead20bab8b0451 (127.6.204.196) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } } }
I20240502 08:04:56.674118 12963 raft_consensus.cc:2932] T 7fe7134eaf2e4e29ac762afe8855bdac P 7e739706a3cb4269b73b3dc63e28ccfe [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 701bb8dd0643469ab2ead20bab8b0451 (127.6.204.196) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } } }
I20240502 08:04:56.675814 12606 raft_consensus.cc:2932] T 7fe7134eaf2e4e29ac762afe8855bdac P ca3300cf09c04d6cac79551a5ab409ea [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 701bb8dd0643469ab2ead20bab8b0451 (127.6.204.196) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } } }
@ 0x7f5b05e18da5 std::__1::__function::__value_func<>::operator()() at ??:0
I20240502 08:04:56.681854 12418 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7fe7134eaf2e4e29ac762afe8855bdac with cas_config_opid_index 2: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:04:56.693286 12429 catalog_manager.cc:5484] T 7fe7134eaf2e4e29ac762afe8855bdac P 7e739706a3cb4269b73b3dc63e28ccfe reported cstate change: config changed from index 2 to 3, NON_VOTER 701bb8dd0643469ab2ead20bab8b0451 (127.6.204.196) evicted. New cstate: current_term: 1 leader_uuid: "b79fbd9f226f4270a50e10dce97bafad" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } } }
I20240502 08:04:56.700490 12862 tablet_service.cc:1514] Processing DeleteTablet for tablet 90b6d92c72f44d89a54c7b1b55470f4b with delete_type TABLET_DATA_TOMBSTONED (TS 701bb8dd0643469ab2ead20bab8b0451 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:47696
I20240502 08:04:56.712328 12861 tablet_service.cc:1514] Processing DeleteTablet for tablet 7fe7134eaf2e4e29ac762afe8855bdac with delete_type TABLET_DATA_TOMBSTONED (TS 701bb8dd0643469ab2ead20bab8b0451 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:47696
W20240502 08:04:56.718112 12419 catalog_manager.cc:4797] TS 701bb8dd0643469ab2ead20bab8b0451 (127.6.204.196:37157): delete failed for tablet 90b6d92c72f44d89a54c7b1b55470f4b because the tablet was not found. No further retry: Not found: Tablet not found: 90b6d92c72f44d89a54c7b1b55470f4b
W20240502 08:04:56.719368 12419 catalog_manager.cc:4797] TS 701bb8dd0643469ab2ead20bab8b0451 (127.6.204.196:37157): delete failed for tablet 7fe7134eaf2e4e29ac762afe8855bdac because the tablet was not found. No further retry: Not found: Tablet not found: 7fe7134eaf2e4e29ac762afe8855bdac
@ 0x7f5b05e18bda std::__1::function<>::operator()() at ??:0
I20240502 08:04:56.728471 12682 consensus_queue.cc:237] T bde9453a83b8464899af354b5e24806f P b79fbd9f226f4270a50e10dce97bafad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } }
I20240502 08:04:56.735766 12606 consensus_queue.cc:237] T 90b6d92c72f44d89a54c7b1b55470f4b P ca3300cf09c04d6cac79551a5ab409ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 } } peers { permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 } } peers { permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "701bb8dd0643469ab2ead20bab8b0451" member_type: NON_VOTER last_known_addr { host: "127.6.204.196" port: 37157 } attrs { promote: true } }
I20240502 08:04:56.738629 12609 raft_consensus.cc:1261] T bde9453a83b8464899af354b5e24806f P ca3300cf09c04d6cac79551a5ab409ea [term 1 FOLLOWER]: Refusing update from remote peer b79fbd9f226f4270a50e10dce97bafad: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:04:56.739187 12963 raft_consensus.cc:1261] T bde9453a83b8464899af354b5e24806f P 7e739706a3cb4269b73b3dc63e28ccfe [term 1 FOLLOWER]: Refusing update from remote peer b79fbd9f226f4270a50e10dce97bafad: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:04:56.740530 13029 consensus_queue.cc:1035] T bde9453a83b8464899af354b5e24806f P b79fbd9f226f4270a50e10dce97bafad [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca3300cf09c04d6cac79551a5ab409ea" member_type: VOTER last_known_addr { host: "127.6.204.194" port: 39897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 08:04:56.742081 12785 consensus_queue.cc:1035] T bde9453a83b8464899af354b5e24806f P b79fbd9f226f4270a50e10dce97bafad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e739706a3cb4269b73b3dc63e28ccfe" member_type: VOTER last_known_addr { host: "127.6.204.193" port: 32777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240502 08:04:56.744910 12575 consensus_peers.cc:485] T 90b6d92c72f44d89a54c7b1b55470f4b P ca3300cf09c04d6cac79551a5ab409ea -> Peer 701bb8dd0643469ab2ead20bab8b0451 (127.6.204.196:37157): Couldn't send request to peer 701bb8dd0643469ab2ead20bab8b0451. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 90b6d92c72f44d89a54c7b1b55470f4b. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:04:56.747566 12682 raft_consensus.cc:1261] T 90b6d92c72f44d89a54c7b1b55470f4b P b79fbd9f226f4270a50e10dce97bafad [term 1 FOLLOWER]: Refusing update from remote peer ca3300cf09c04d6cac79551a5ab409ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:04:56.749127 13036 consensus_queue.cc:1035] T 90b6d92c72f44d89a54c7b1b55470f4b P ca3300cf09c04d6cac79551a5ab409ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "b79fbd9f226f4270a50e10dce97bafad" member_type: VOTER last_known_addr { host: "127.6.204.195" port: 46407 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
@ 0x7f5afcb86937 kudu::Thread::SuperviseThread() at ??:0